Implementation:Spotify Luigi Retcodes

Overview

The luigi.retcodes module defines the exit code logic for the luigi CLI command. It provides the retcode configuration class for customizing which exit codes are returned for various failure scenarios, and the run_with_retcodes function which serves as the main entry point for the Luigi binary. This module enables programmatic detection of whether Luigi completed its tasks successfully and, if not, what category of failure occurred.

Source Location

Property	Value
Source File	luigi/retcodes.py
Lines of Code	108
Module	luigi.retcodes
Domain	CLI, Process_Management

Import Statement

from luigi.retcodes import retcode, run_with_retcodes

Class: retcode

retcode(luigi.Config)

Configuration class for exit codes. Values are read from the [retcode] section in luigi.cfg.

Parameters

Parameter	Type	Default	Description
unhandled_exception	IntParameter	4	Exit code for internal Luigi errors (uncaught exceptions before or during setup).
missing_data	IntParameter	0	Exit code when there are incomplete ExternalTask dependencies (still_pending_ext status).
task_failed	IntParameter	0	Exit code when a task's run() method fails (failed status).
already_running	IntParameter	0	Exit code when another instance is already running (applies to both local --lock and luigid lock, or run_by_other_worker status).
scheduling_error	IntParameter	0	Exit code when a task's complete() or requires() fails, or the task limit is reached (scheduling_error status).
not_run	IntParameter	0	Exit code when a task is not granted run permission by the scheduler (not_run status), or when the root task is neither in completed nor already_done.

Note: The default values of 0 for most parameters mean that by default, Luigi exits with code 0 even on failure. This is for backwards compatibility. To enable meaningful exit codes, configure non-zero values in luigi.cfg.

Function: run_with_retcodes

run_with_retcodes(argv)

Parameter	Type	Description
argv	list	Command-line arguments, conceptually sys.argv[1:].

This is the main entry point for the luigi CLI command. It runs the Luigi pipeline and calls sys.exit() with an appropriate return code based on the execution outcome.

Execution Flow

1. Parses the command line with CmdlineParser.global_instance(argv) to read the [retcode] configuration.
2. Calls luigi.interface._run(argv) to execute the pipeline, obtaining a worker object.
3. If a PidLockAlreadyTakenExit is raised, exits with retcodes.already_running.
4. If any other unhandled exception occurs, sets up logging and exits with retcodes.unhandled_exception.
5. After execution, builds the execution summary by calling luigi.execution_summary._summary_dict(worker).
6. Evaluates the following conditions in priority order (exit code is the maximum matching code):

Condition	Checked Status	Exit Code Parameter
External dependencies missing	still_pending_ext	retcodes.missing_data
Tasks failed	failed	retcodes.task_failed
Run by other worker	run_by_other_worker	retcodes.already_running
Scheduling error	scheduling_error	retcodes.scheduling_error
Not run	not_run	retcodes.not_run

If the computed exit code is 0 but the root task is not in completed or already_done, the function exits with retcodes.not_run.

Configuration

[retcode]
unhandled_exception = 4
missing_data = 1
task_failed = 2
already_running = 3
scheduling_error = 5
not_run = 6

Usage Example

As CLI Entry Point

The luigi command-line tool invokes run_with_retcodes:

# In a shell script, check the exit code
luigi --module my_tasks MyTask --date 2024-01-15
exit_code=$?

if [ $exit_code -eq 0 ]; then
    echo "Pipeline completed successfully"
elif [ $exit_code -eq 2 ]; then
    echo "A task failed"
elif [ $exit_code -eq 1 ]; then
    echo "Missing external data"
fi

Programmatic Usage

from luigi.retcodes import run_with_retcodes

# This will call sys.exit() with the appropriate return code
run_with_retcodes(['--module', 'my_tasks', 'MyTask', '--date', '2024-01-15'])

Exit Code Priority

When multiple failure conditions are present, the maximum configured exit code is used. For example, if both task_failed (code 2) and missing_data (code 1) conditions are true, the exit code will be 2.

The priority calculation is:

expected_ret_code = max(code * (1 if condition else 0) for code, condition in codes_and_conds)

Dependencies

• Luigi core: luigi, luigi.IntParameter, luigi.setup_logging.InterfaceLogging, luigi.cmdline_parser.CmdlineParser, luigi.interface, luigi.execution_summary
• Python standard library: sys, logging

Related Principles

• Spotify_Luigi_Pipeline_Execution

See Also

• Spotify_Luigi_Event_Constants - Events that influence execution summary
• luigi.execution_summary - Builds the summary dict used for exit code determination
• luigi.interface._run - The core pipeline execution function