Principle:Vespa engine Vespa Control File Initialization

Control File Initialization

Logging Observability

Overview

Control file initialization creates or opens a memory-mapped file that serves as the shared-memory communication channel between log-producing processes and the runtime level control tool (vespa-logctl). The file has a versioned header, an application prefix section, and per-component level entries. This mechanism enables zero-cost level checking and runtime reconfiguration without process restart.

Motivation

In a distributed system like Vespa with hundreds of processes, operators frequently need to increase logging verbosity for a specific component to diagnose issues, then restore normal levels. Restarting the process to change a log configuration flag would be disruptive and slow.

The control file approach solves this by:

1. Using memory-mapped files so that level checks are simple memory reads with no system call overhead.
2. Allowing an external tool (vespa-logctl) to modify the mapped file while the process is running.
3. Supporting per-component granularity so that only the relevant component becomes verbose.
4. Providing a fallback default so that components without explicit entries use the default level string.

File Format

The control file has the following binary layout:

+-------------------------------------------+
| Header: "Vespa log control file version 1\n" |
| (34 bytes, null-padded to alignment)       |
+-------------------------------------------+
| Application Prefix (up to 64 bytes)        |
+-------------------------------------------+
| Component Entry 0                          |
|   - Component name (null-terminated)       |
|   - Level bitmask (8 levels, 1 byte each)  |
+-------------------------------------------+
| Component Entry 1                          |
+-------------------------------------------+
| ...                                        |
+-------------------------------------------+

Header

The header string is exactly "Vespa log control file version 1\n". This is validated when opening an existing file to ensure compatibility.

Application Prefix

The application prefix (up to 64 characters, defined by the maxPrefix constant) identifies the Vespa application that owns the log control file. It is used as a prefix in component names within the file.

Component Entries

Each component entry maps a component name (typically a Java logger name or C++ component identifier) to a level bitmask. The bitmask has one byte per level, with 8 levels defined:

Index	Level	Default (all -debug -spam)
0	fatal	ON
1	error	ON
2	warning	ON
3	config	ON
4	info	ON
5	event	ON
6	debug	OFF
7	spam	OFF

Memory Mapping

The control file is opened with RandomAccessFile in read-write mode and mapped into memory using MappedByteBuffer. This provides:

• Zero-copy access: Level checks read directly from the memory-mapped region without any I/O system calls.
• Shared visibility: When vespa-logctl modifies a level byte in the file, the change is immediately visible to the process through the memory mapping.
• Persistence: The file survives process crashes, so levels are restored on restart.

Initialization Sequence

The control file initialization follows this sequence:

1. Resolve the file path: Derived from VESPA_LOG_CONTROL_DIR + VESPA_SERVICE_NAME + .logcontrol, or from an explicit VESPA_LOG_CONTROL_FILE.
2. Open or create the file: If the file does not exist, it is created with the versioned header and application prefix. If it exists, the header is validated.
3. Memory-map the file: The file is mapped into the process address space using FileChannel.map().
4. Create the level controller repository: A MappedLevelControllerRepo wraps the mapped buffer and provides component-level lookups.
5. Register a default level controller: A DefaultLevelController is created from the log level string to handle components that have no explicit entry in the file.

Fallback Behavior

If the control file path is null (because no control directory or service name was provided), a default-only level controller is used. This controller applies the same levels to all components based on the VESPA_LOG_LEVEL string. Runtime level changes are not possible in this mode.

Design Constraints

• Maximum prefix length: The application prefix is limited to 64 characters (maxPrefix = 64). Longer prefixes are truncated.
• Fixed number of levels: The file format supports exactly 8 levels (numLevels = 8). Adding new levels would require a new file format version.
• File locking: The control file does not use explicit file locking. Concurrent access safety relies on the atomic nature of single-byte writes in memory-mapped files.
• Stale entries: Component entries are never removed from the file during the lifetime of a process. Old entries from previous runs may remain but are harmless.

Interaction with Other Components

• Log Target and Level Configuration: Provides the control file path and default level string used during initialization.
• Logger Handler Installation: The level controller repository created here is passed to the VespaLogHandler.
• Runtime Level Control: The vespa-logctl tool reads and writes the same memory-mapped file to change levels at runtime.

References

• VespaLevelControllerRepo.java on GitHub
• control-file.cpp on GitHub

Related Pages

• Implementation:Vespa_engine_Vespa_VespaLevelControllerRepo_Constructor
• Heuristic:Vespa_engine_Vespa_Log_Level_Inheritance_Polling