Implementation:Spotify Luigi LuigiConfigParser

Overview

Concrete tool for centralized configuration management across all pipeline components provided by Luigi.

Description

LuigiConfigParser is Luigi's primary configuration engine, built on Python's standard ConfigParser. It reads INI-style configuration files (with .cfg, .conf, or .ini extensions) from a predefined search path and exposes settings to every Luigi component through a singleton instance.

The parser extends standard ConfigParser with several Luigi-specific behaviors:

Combined interpolation: Values are processed through a two-stage interpolation pipeline -- first BasicInterpolation for %(key)s-style references, then EnvironmentInterpolation for ${ENVVAR}-style environment variable substitution.
Case-preserving keys: Unlike default ConfigParser behavior which lowercases option names, LuigiConfigParser sets optionxform = str to preserve the original casing of configuration keys.
Dash-to-underscore normalization: Options specified with dashes (e.g., retry-count) are transparently resolved to their underscore equivalents (retry_count), with a deprecation warning guiding users toward the canonical underscore style.
Default value support: All typed accessor methods (get, getint, getfloat, getboolean) accept an optional default parameter, allowing callers to provide fallback values when a section or option is missing.
Auto-section creation: The set() method automatically creates a configuration section if it does not already exist, simplifying programmatic configuration.

The companion module luigi.configuration.core provides the public API through two functions: get_config() returns the singleton parser instance, and add_config_path() appends additional configuration files and triggers a reload.

Usage

Use LuigiConfigParser when:

You need to read or set Luigi configuration values programmatically within task code or custom components.
You want to inject additional configuration files beyond the default search paths (e.g., environment-specific overrides).
You are building Luigi extensions that need access to framework-wide settings.

Code Reference

Source Location

File	Lines	Role
`luigi/configuration/core.py`	L56-93	Public API: `get_config()`, `add_config_path()`
`luigi/configuration/cfg_parser.py`	L117-216	`LuigiConfigParser` class definition
`luigi/configuration/base_parser.py`	L23-41	`BaseParser` with singleton pattern and `add_config_path()`

Signature

# luigi/configuration/core.py
def get_config(parser=None):
    """Get configs singleton for parser."""

def add_config_path(path):
    """Select config parser by file extension and add path into parser."""

# luigi/configuration/cfg_parser.py
class LuigiConfigParser(BaseParser, ConfigParser):
    NO_DEFAULT = object()
    enabled = True
    _config_paths = [
        '/etc/luigi/client.cfg',
        '/etc/luigi/luigi.cfg',
        'client.cfg',
        'luigi.cfg',
    ]

    def get(self, section, option, default=NO_DEFAULT, **kwargs): ...
    def getboolean(self, section, option, default=NO_DEFAULT): ...
    def getint(self, section, option, default=NO_DEFAULT): ...
    def getfloat(self, section, option, default=NO_DEFAULT): ...
    def getintdict(self, section): ...
    def set(self, section, option, value=None): ...

Import

from luigi.configuration import get_config, add_config_path
# or
from luigi.configuration.cfg_parser import LuigiConfigParser

I/O Contract

Inputs

Parameter	Type	Description
`parser` (for `get_config`)	`str` or `None`	Parser name: `'cfg'`, `'conf'`, `'ini'`, or `'toml'`. Defaults to `LUIGI_CONFIG_PARSER` env var or `'cfg'`.
`path` (for `add_config_path`)	`str`	Filesystem path to an additional configuration file.
`section`	`str`	Configuration section name (e.g., `'scheduler'`, `'core'`).
`option`	`str`	Configuration option name within the section.
`default`	varies	Fallback value returned when the section/option is missing.

Outputs

Method	Return Type	Description
`get_config()`	`LuigiConfigParser`	Singleton configuration parser instance.
`add_config_path()`	`bool`	`True` if the file was found and added, `False` otherwise.
`get()`	`str`	String value of the configuration option.
`getint()`	`int`	Integer value of the configuration option.
`getfloat()`	`float`	Float value of the configuration option.
`getboolean()`	`bool`	Boolean value of the configuration option.
`getintdict()`	`dict[str, int]`	All key-value pairs in a section, with values cast to integers.

Usage Examples

Reading Configuration Values

from luigi.configuration import get_config

config = get_config()

# Read scheduler state path with a default
state_path = config.get('scheduler', 'state_path', '/var/lib/luigi-server/state.pickle')

# Read retry count as integer
retry_count = config.getint('scheduler', 'retry_count', 999999999)

# Read a boolean flag
record_history = config.getboolean('scheduler', 'record_task_history', False)

Adding a Custom Configuration File

from luigi.configuration import add_config_path

# Add an environment-specific override file
added = add_config_path('/etc/luigi/production.cfg')
if added:
    print("Production configuration loaded")

Setting Configuration Values Programmatically

from luigi.configuration import get_config

config = get_config()

# Override scheduler state path at runtime
# Automatically creates the [scheduler] section if it does not exist
config.set('scheduler', 'state_path', '/tmp/luigi-state.pickle')

Using Environment Variable Interpolation

Create a luigi.cfg file that references environment variables:

[task_history]
db_connection = postgresql://${DB_USER}:${DB_PASSWORD}@${DB_HOST}/luigi

[scheduler]
state_path = ${LUIGI_STATE_DIR}/state.pickle

Then set the environment variables before running Luigi:

export DB_USER=luigi
export DB_PASSWORD=secret
export DB_HOST=db.example.com
export LUIGI_STATE_DIR=/var/lib/luigi-server
luigid

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment