Principle:Spotify Luigi Task Simulation
| Knowledge Sources | |
|---|---|
| Domains | Testing, Development |
| Last Updated | 2026-02-10 08:00 GMT |
Overview
Enabling tasks to re-execute on every run regardless of previous completion state for development and debugging purposes.
Description
Task simulation is the practice of configuring a pipeline task's output target to always report itself as incomplete, thereby forcing the task to execute on every pipeline run regardless of whether its output already exists. In normal pipeline operation, the orchestrator checks whether a task's output target exists and skips execution if it does, implementing the idempotency guarantee. During development and debugging, however, this behavior is counterproductive: developers need to re-run tasks repeatedly to test changes to task logic without manually deleting output files between runs. Task simulation provides a special target type that wraps a real target but overrides the existence check to always return false, causing the orchestrator to treat the task as incomplete and execute it every time.
Usage
Use task simulation during development and debugging when iterating on task logic and needing to re-run tasks without manual cleanup. It is also useful for tasks that should always run regardless of prior state, such as notification tasks, health checks, or tasks whose side effects (sending emails, triggering webhooks) should occur on every pipeline execution.
Theoretical Basis
Task simulation operates by overriding the completion predicate in the pipeline's execution model:
1. Standard Completion Model -- In normal operation, the pipeline's execution decision for a task follows:
IF output_target.exists() = TRUE THEN skip execution
ELSE execute task
This is the foundation of idempotent pipeline execution.
2. Simulated Target -- A simulation target wraps a real target and overrides the existence check:
SimulatedTarget.exists() -> FALSE (always)
All other operations (open, read, write) delegate to the wrapped real target, so the task can still produce actual output.
3. Execution Guarantee -- Because exists() always returns false, the orchestrator will always schedule and execute the task:
FOR EVERY pipeline run:
output.exists() = FALSE (by override)
THEREFORE task executes
task writes to wrapped real target (actual I/O still occurs)
4. Decorator Pattern -- The simulation target follows the decorator pattern: it has the same interface as a regular target but modifies one specific behavior (existence check) while preserving all others. This means:
* Writing to a simulated target writes to real storage
* Reading from a simulated target reads from real storage
* Only the existence check is altered
5. Scope Control -- Simulation is applied at the individual task level, not globally. Only tasks whose output is explicitly wrapped in a simulation target are affected. All other tasks in the pipeline continue to use standard completion checking, maintaining pipeline efficiency.
6. Development vs. Production -- A common pattern is conditional application:
IF environment = development THEN use SimulatedTarget
ELSE use real target
This prevents accidental re-execution in production while enabling rapid iteration in development.
The key insight is that pipeline execution semantics are entirely determined by the target's existence check, making it a clean extension point for altering execution behavior without modifying task logic or the orchestrator.