Heuristic:Vespa engine Vespa Log Level Inheritance Polling
| Knowledge Sources | |
|---|---|
| Domains | Logging, Optimization |
| Last Updated | 2026-02-09 00:00 GMT |
Overview
Log level runtime control via mmap-shared files with 2999ms polling interval, component-level inheritance, and the default "all -debug -spam" level string.
Description
Vespa uses memory-mapped control files shared between Java and C++ processes to enable runtime log level changes without process restarts. A background task polls the control file every ~3 seconds (2999ms period, 1000ms initial delay). Log levels inherit hierarchically: a component (e.g., "foo.bar") inherits from its parent ("foo"), which inherits from the "default" entry. The slightly non-round polling interval (2999ms instead of 3000ms) avoids synchronized polling ("herding") across many Java processes.
Usage
Apply this heuristic when debugging log level changes not taking effect (wait up to 3 seconds), understanding the log level hierarchy, or tuning the default log level. The default all -debug -spam enables INFO and above, which is the recommended baseline for production.
The Insight (Rule of Thumb)
- Action 1: Log level changes propagate within ~3 seconds (2999ms polling period).
- Action 2: The polling period is intentionally 2999ms (not 3000ms) to avoid synchronized checks across processes.
- Action 3: Component levels inherit: "foo.bar" inherits from "foo", which inherits from "default".
- Action 4: Default level string is
all -debug -spam(INFO, WARNING, CONFIG, EVENT enabled; DEBUG, SPAM disabled). - Action 5: Each level is stored as 4-byte integer (" ON" = 0x20204f4e, " OFF" = 0x204f4646) for atomic reads.
- Action 6: ZooKeeper/Curator logs are automatically filtered to separate rolling files (10 files x 10MB = 100MB max).
- Trade-off: 3-second polling delay vs. immediate level change notification; avoids file-watch complexity.
Reasoning
Memory-mapped files provide zero-copy sharing between Java and C++ processes without IPC overhead. The 2999ms interval prevents "thundering herd" effects where hundreds of Vespa processes all check the control file at the same moment. Component inheritance reduces configuration burden: setting a level on "com.yahoo" automatically applies to all sub-components. The 4-byte integer encoding enables atomic reads without locking. The ZooKeeper log filter prevents verbose ZK client logs from overwhelming the main Vespa log.
Code Evidence
Polling interval from VespaLevelControllerRepo.java:
private void openCtlFile() {
if (checkBackRunner == null) {
checkBackRunner = new CheckBackRunner();
LogSetup.getTaskRunner().schedule(checkBackRunner, 1000, 2999);
// ^delay ^period (ms)
}
}
Default level from control-file.cpp:
unsigned int* ControlFile::defaultLevels() {
static unsigned int levels[Logger::NUM_LOGLEVELS + 1];
const char *env = getenv("VESPA_LOG_LEVEL");
if (!env) {
env = "all -debug -spam"; // Default: all levels except debug/spam
}
makeLogLevelArray(levels, sizeof levels, env);
return levels;
}
Level encoding from MappedLevelController.java:
private static final int ONVAL = 0x20204f4e; // " ON" in ASCII
private static final int OFFVAL = 0x204f4646; // " OFF" in ASCII
ZooKeeper filter from LogSetup.java:
static class ZooKeeperFilter implements Filter {
private static final int FILE_SIZE = 10*1024*1024; // 10 MB per file
private static final int maxFilesCount = 10; // 10 files max = 100MB total
}