Principle:Iterative Dvc Plot Data Source Resolution
| Knowledge Sources | |
|---|---|
| Domains | Visualization, Version_Control |
| Last Updated | 2026-02-10 00:00 GMT |
Overview
Plot data source resolution is the process of iterating over Git revisions to collect plot definitions and their associated data sources from each revision's filesystem, enabling cross-revision comparison of metrics and visualizations.
Description
In version-controlled data science projects, one of the most powerful visualization capabilities is comparing metrics across multiple Git revisions: different experiments, branches, or tagged releases. Plot data source resolution addresses the fundamental challenge of gathering plot data from multiple historical states of a repository, where each revision may have different file contents, different pipeline configurations, and even different sets of tracked plots.
The resolution process works by iterating over a list of requested revisions, switching the repository's virtual filesystem to each revision in turn, and collecting both the plot definitions and their data sources at that point in history. Rather than eagerly loading all data files immediately, the process employs a lazy evaluation strategy: for each data source, it records a callable (a partially applied function) that, when invoked, will read and parse the actual file contents. This deferred loading is critical for performance because plot commands may reference many data files across many revisions, and not all of them may ultimately be needed for the final visualization.
The workspace (uncommitted local state) is treated as a special pseudo-revision that is always processed first when included. This ensures that the user's current working state serves as the baseline for comparison. Subsequent revisions are resolved through Git's object storage, with the repository's filesystem temporarily switched to a GitFileSystem backed by each commit's tree.
Usage
Use plot data source resolution when:
- Building visualizations that compare metrics across multiple Git commits, branches, or tags (e.g., dvc plots diff HEAD~3 HEAD~1 HEAD).
- The workspace must be included alongside historical revisions as a comparison baseline.
- Data loading must be deferred to avoid unnecessary I/O when only a subset of collected sources will actually be rendered.
- The system must gracefully handle revisions where certain plot files do not yet exist or have different schemas.
Theoretical Basis
The multi-revision collection algorithm follows a generator pattern that yields per-revision results lazily:
FUNCTION collect(repo, targets, revisions, props):
IF revisions is None:
revisions = ["workspace"]
ELSE:
// Ensure workspace is processed first if present
IF "workspace" in revisions:
MOVE "workspace" to front of revisions
FOR each rev in revisions:
WITH switch_repo(repo, rev):
// Phase 1: Collect definitions at this revision
definitions = collect_definitions(repo, targets, props)
IF definitions exist:
result[rev]["definitions"] = definitions
// Phase 2: Infer data file targets from definitions
data_targets = infer_data_sources(definitions)
// Phase 3: Build lazy data source callables
sources = {}
FOR each plot_file in find_plot_outputs(data_targets):
sources[plot_file] = {
"props": plot_file.properties UNION props,
"data_source": PARTIAL(parse, fs, plot_file.path, props)
}
result[rev]["sources"] = sources
YIELD result
The key design principles are:
Lazy data loading: Each data source is represented as a callable (functools.partial) rather than loaded data. The callable captures the filesystem, path, and properties needed to load the file. This allows the consumer to decide when and whether to actually read the data, enabling parallel resolution and selective loading.
Filesystem switching: Rather than checking out files to disk, the system uses virtual filesystem abstractions. The switch_repo context manager swaps the repository's filesystem to a GitFileSystem for historical revisions, allowing all file operations to transparently read from Git's object store. This avoids modifying the working directory and supports concurrent access.
Generator-based iteration: The collection process yields results one revision at a time rather than accumulating all revisions in memory. This is important for large comparison sets (e.g., all branches or all experiments) where holding all data simultaneously would be memory-prohibitive.