Principle:Iterative Dvc Stage Freshness Detection
| Knowledge Sources | |
|---|---|
| Domains | Pipeline_Management, Change_Detection |
| Last Updated | 2026-02-10 00:00 GMT |
Overview
Stage freshness detection is the process of determining whether a pipeline stage needs re-execution by comparing its current state against its recorded state at multiple levels: stage definition, dependency checksums, and output checksums.
Description
A core requirement for efficient pipeline reproduction is the ability to skip stages that have not changed since their last execution. Naive approaches -- such as always re-running every stage -- waste computational resources, while overly aggressive caching can produce stale results. Stage freshness detection solves this by implementing a multi-level change detection strategy that examines three distinct dimensions of a stage's state.
The first level is stage definition change detection. For pipeline stages, this compares the current command string against the command recorded in the lockfile. If the user has modified the command (e.g., changed training hyperparameters in the command line), the stage is considered changed regardless of whether its data dependencies have changed. For single-stage DVC files, an MD5 hash of the stage definition is used instead.
The second level is dependency change detection. Each dependency (input file, parameter file, etc.) maintains a recorded checksum from the lockfile. The detection process compares the current state of each dependency against its recorded state. This includes special handling for parameter dependencies (where individual parameter values are tracked), dataset dependencies, and the ability to tolerate missing dependencies when upstream stages are expected to produce them.
The third level is output change detection. Similarly to dependencies, each output has a recorded checksum. If an output has been modified externally, deleted, or is not present in the cache, the stage is marked as changed. This catches cases where outputs have been manually altered or corrupted.
Two special stage types bypass the normal detection logic: callback stages (stages with no dependencies or outputs) are always considered changed, and always_changed stages (explicitly flagged by the user) unconditionally require re-execution. Frozen stages are the opposite -- they are always considered fresh, blocking change propagation through them.
Usage
Stage freshness detection is essential whenever:
- Incremental pipeline reproduction is needed -- only changed stages should be re-executed.
- Pipeline status reporting is desired -- users need to know which stages are out of date and why.
- Cost optimization is important -- expensive computations (model training, data processing) should be skipped when inputs have not changed.
- Correctness guarantees are required -- all three levels must agree that a stage is fresh before skipping execution.
The short-circuit evaluation order (stage definition first, then dependencies, then outputs) is intentional: stage definition checks are computationally cheap (string comparison), dependency checks are moderately expensive (file hashing), and output checks are the most expensive (potentially hashing large output files). By checking in this order, the system fails fast on the cheapest signals.
Theoretical Basis
The multi-level change detection algorithm follows a short-circuit evaluation pattern:
PROCEDURE StageChanged(stage, allow_missing, upstream):
// Level 1: Stage definition change (cheapest check)
IF stage IS PipelineStage:
stage_changed = (stage.cmd != lockfile.cmd) // cmd_changed flag
ELSE:
stage_changed = (stage.md5 != COMPUTE_MD5(stage))
IF stage_changed:
RETURN True
// Level 2: Dependency change detection
IF stage.frozen:
deps_changed = False // frozen stages don't check deps
ELSE IF stage.is_callback OR stage.always_changed:
deps_changed = True // always considered changed
ELSE:
FOR EACH dep IN stage.deps:
status = dep.status() // compare current vs recorded checksum
IF status INDICATES change:
IF allow_missing AND status == "deleted":
// Check if upstream will recreate this dep
IF upstream HAS matching output with different hash:
deps_changed = True // upstream changed it
ELSE:
CONTINUE // tolerate missing, upstream unchanged
ELSE:
deps_changed = True
BREAK
deps_changed = False // all deps unchanged
IF deps_changed:
RETURN True
// Level 3: Output change detection (most expensive)
FOR EACH out IN stage.outs:
status = out.status()
IF status INDICATES change:
IF allow_missing AND status IN ["deleted", "not in cache"]:
CONTINUE // tolerate missing outputs
ELSE:
RETURN True
RETURN False // stage is fresh, can be skipped
Key theoretical properties:
- Short-circuit evaluation: The three levels are checked in order of increasing computational cost. As soon as any level detects a change, the remaining levels are skipped.
- Monotonic change propagation: If a stage is changed, all of its downstream dependents will also detect changes (via their dependency check), ensuring transitive correctness.
- Allow-missing tolerance: The allow_missing parameter enables partial pipeline execution where some stages may not have produced their outputs yet. The upstream parameter further refines this by checking whether a missing dependency's upstream producer has actually changed, preventing false negatives.
- Frozen boundary: Frozen stages act as change propagation barriers -- changes to their dependencies are not detected, preventing downstream re-execution.