Workflow:Vespa engine Vespa Logging framework initialization
| Knowledge Sources | |
|---|---|
| Domains | Logging, Observability, Infrastructure |
| Last Updated | 2026-02-09 12:00 GMT |
Overview
End-to-end process for initializing and configuring the Vespa logging framework across both Java and C++ components, with runtime log level control via memory-mapped files.
Description
This workflow describes how Vespa's unified logging framework is initialized and operates across the dual C++/Java architecture. Both languages share a common tab-delimited log format and a memory-mapped control file mechanism for runtime log level changes. The Java side integrates with java.util.logging through VespaLogHandler and VespaFormatter, while the C++ side uses the LOG macro with a static Logger instance. The VespaLevelControllerRepo manages the control file that enables operators to change log levels at runtime without restarting services.
Usage
Execute this workflow when initializing any Vespa service (container, content node, config server) or when implementing custom logging integration. This is relevant whenever you need to understand how log output is produced, formatted, and controlled at runtime. Use this to diagnose logging configuration issues, implement custom log targets, or understand the control file mechanism for production log level management.
Execution Steps
Step 1: Log target and level configuration
Determine the log output target and initial log levels from configuration sources. Java reads from system properties (vespa.log.target, vespa.log.level) with fallback to environment variables (VESPA_LOG_TARGET, VESPA_LOG_LEVEL) and defaults (stderr, "all -debug -spam"). C++ reads VESPA_LOG_TARGET to select between file descriptor, file, or stderr output. The service name is resolved from VESPA_SERVICE_NAME.
Key considerations:
- Java configuration priority: system properties, then environment variables, then defaults
- Log targets: "fd:2" (stderr), "fd:1" (stdout), or "file:/path/to/logfile"
- Default level string "all -debug -spam" enables all levels except debug and spam
- The service name identifies the process in log messages and control file paths
Step 2: Control file initialization
Create or open the memory-mapped log control file that enables runtime level changes. The control file is located at {logControlDir}/{serviceName}.logcontrol and contains a 108-byte header followed by per-component level entries. Java's VespaLevelControllerRepo creates the file and writes the header containing the file format version and application prefix. C++ opens the same file via mmap for shared access.
Key considerations:
- The header format is exactly: "Vespa log control file version 1\nPrefix: {prefix}\n" padded to 108 bytes
- Both C++ and Java processes can read/write the same control file via mmap
- Component entries are 4-byte aligned for efficient memory-mapped access
- The control file directory defaults to VESPA_LOG_CONTROL_DIR
Step 3: Logger handler installation
Install the Vespa log handler into the logging framework. In Java, LogSetup.initVespaLogging clears all existing handlers, sets the root logger to Level.ALL, and installs VespaLogHandler as the sole handler. The handler connects to the log target, VespaLevelControllerRepo (for level control), and VespaFormatter (for output formatting). In C++, the LOG_SETUP macro creates a static Logger instance per component.
Key considerations:
- Java clears all existing handlers to prevent duplicate output
- Root logger is set to ALL; actual filtering happens in VespaLogHandler
- VespaLogHandler is synchronized for thread-safe log publishing
- C++ Logger is a static instance initialized once per component via LOG_SETUP
Step 4: Log message formatting and output
When a log statement executes, first check if the level is enabled for the component (via control file lookup). If enabled, format the message in Vespa's common tab-delimited format: timestamp, hostname, pid/threadid, service name, component, level, message. Special characters are escaped (newlines as \n, tabs as \t, backslashes as \\). The formatted message is written to the configured log target.
Key considerations:
- Level checking happens before formatting to avoid unnecessary work
- Java VespaFormatter adds exception stack traces with proper escaping
- EVENT level messages receive special serialization for structured event data
- Specific verbose loggers (Jetty, SPI Fly) have their levels reduced automatically
Step 5: Runtime level control
Operators modify log levels at runtime using the vespa-logctl command-line tool, which writes to the memory-mapped control file. Java's CheckBackRunner timer task (running every 3 seconds) detects changes by rereading the mmap'd file and updates all logger instances. C++ reads the control file directly on each log check. Component levels follow hierarchical inheritance: child components inherit parent levels via dot-notation naming.
Key considerations:
- Level changes take effect within 3 seconds on the Java side (CheckBackRunner interval)
- C++ level checks read mmap'd memory directly, so changes are near-instant
- Hierarchical inheritance means setting "myapp" level affects "myapp.component" and below
- Eight log levels are supported: fatal, error, warning, info, config, debug, spam, event
Step 6: Log rotation and special handling
Handle log file rotation and special log routing. File-based log targets support rotation when the output file is moved or recreated. ZooKeeper and Curator library logs are optionally routed to a separate rotating log file (10 files of 10MB each) to prevent flooding the main log. The RejectFilter suppresses known noisy messages from specific components.
Key considerations:
- ZooKeeper log separation is configured via vespa.log.zookeeper.file.prefix property
- File rotation is detected by checking if the file descriptor still points to the expected path
- RejectFilter matches on exact message prefix strings for suppression
- The buffered logger (C++ side) deduplicates repeated messages within a time window