Principle:Iterative Dvc Experiment Pipeline Execution

Knowledge Sources	DVC Documentation
Domains	Experiment_Management, Pipeline_Execution
Last Updated	2026-02-10 00:00 GMT

Overview

Experiment pipeline execution is the process of running a defined ML pipeline within an isolated experiment context, capturing all inputs and outputs under version control for reproducibility.

Description

Machine learning workflows are typically structured as multi-stage pipelines: data ingestion, feature engineering, model training, evaluation, and so on. When these pipelines are executed as experiments, additional requirements arise beyond simple script execution. The pipeline must run against a specific set of parameters (potentially different from the current workspace), all outputs must be captured and versioned, and the execution must be attributable to a specific experiment identity.

Experiment pipeline execution bridges the gap between pipeline definition (what stages to run and in what order) and experiment management (tracking which parameters produced which results). The key insight is that an experiment is not merely a pipeline run -- it is a pipeline run in context. That context includes the baseline revision, parameter overrides, execution environment, and the resulting output artifacts. By making the execution context explicit and version-controlled, experiment pipeline execution enables true reproducibility: any experiment can be re-run from its captured state to produce identical results.

The execution model supports three modes with increasing levels of isolation and scalability. Immediate execution runs the pipeline in the current process, either in the workspace or in a temporary directory. Queued execution places the experiment in a persistent task queue (backed by Celery) for later execution, allowing batch submission of many experiments. Distributed execution extends queued execution by starting worker processes that consume from the queue, enabling parallel experiment runs across multiple workers or machines.

Usage

Use experiment pipeline execution when:

• You need to run a DVC pipeline with specific parameter overrides and track the results as a named experiment
• You want to batch-submit multiple experiments to a queue for sequential or parallel processing
• You need to run experiments in a temporary directory to avoid interfering with the current workspace
• You are building a CI/CD pipeline that triggers experiment runs as part of an automated workflow
• You need to distribute experiment execution across multiple compute nodes

This is the appropriate technique whenever pipeline reproduction needs to be combined with experiment tracking, rather than running pipelines in an ad-hoc manner.

Theoretical Basis

Experiment pipeline execution follows a dispatch-execute-collect pattern determined by the execution mode:

function run_experiment(repo, params, targets, mode):
    # Step 1: Parse and validate parameter overrides
    path_overrides = parse_params(params)

    # Step 2: Check for sweep overrides
    if contains_sweep_overrides(path_overrides):
        if mode != QUEUED:
            raise Error("Sweeps require queued execution")
        sweeps = expand_sweeps(path_overrides)
    else:
        sweeps = [path_overrides]

    # Step 3: Dispatch based on execution mode
    if mode == RUN_ALL:
        # Execute all queued experiments via Celery workers
        return reproduce_celery(jobs=num_workers)

    results = {}
    for sweep_config in sweeps:
        if mode == IMMEDIATE:
            # Prepare workspace and run in-place or tmp_dir
            queue = select_queue(tmp_dir)
            queue.put(targets, params=sweep_config)
            results.update(queue.reproduce())

        elif mode == QUEUED:
            # Push to Celery queue for later execution
            celery_queue.put(targets, params=sweep_config)

    return results

The Celery execution path introduces additional complexity for distributed processing:

function reproduce_celery(entries, jobs):
    start_workers(count=jobs)
    results = {}
    for entry in entries:
        wait_for_start(entry)
        follow_logs(entry)
        result = collect_result(entry)
        if result.success:
            results[result.rev] = result.exp_hash
        else:
            record_failure(entry)
    return results

Key theoretical properties of the execution model:

1. Mode orthogonality: The experiment preparation step is identical regardless of execution mode; only the dispatch mechanism changes
2. Sweep constraint: Sweep overrides (multi-value parameters) are only valid in queued mode because each sweep point becomes a separate queue entry
3. Idempotent reproduction: Given the same stash entry, re-execution produces identical results (assuming deterministic pipeline stages)
4. Fault isolation: Failed experiments in a queue do not block or affect other experiments; failures are recorded separately

Related Pages

Implemented By

• Implementation:Iterative_Dvc_Experiments_Run