Principle:Iterative Dvc Dataset Resolution
| Knowledge Sources | |
|---|---|
| Domains | API, Dataset_Management |
| Last Updated | 2026-02-10 00:00 GMT |
Overview
Dataset resolution is the process of retrieving a dataset's complete definition -- including its type, storage location, and locked version details -- from a version-controlled pipeline specification, providing consumers with all information needed to access the data at a specific point in time.
Description
Data pipelines typically declare their input datasets as named dependencies with type-specific configuration. A dataset might be a URL to a remote file, a path within cloud storage, a database connection string, or a reference to an output of another pipeline stage. Each dataset type carries different location semantics: a URL dataset requires an HTTP endpoint, a storage dataset requires a bucket and key, and a database dataset requires connection parameters and a query. Dataset resolution provides a unified interface for retrieving these heterogeneous definitions in a normalized form.
The resolution process involves two layers of information. The first layer is the declaration, found in the pipeline definition file (e.g., dvc.yaml), which specifies the dataset name, type, and mutable configuration such as a URL pattern or storage path. The second layer is the lock information, found in the lock file (e.g., dvc.lock), which captures the exact state of the dataset at the time of the last successful pipeline execution -- including content hashes, ETags, timestamps, or database query checksums. Together, these two layers provide both the logical definition ("what data source") and the pinned version ("which exact snapshot").
By combining declaration and lock data into a single resolved record, dataset resolution enables several critical workflows. Consumers can reproduce a previous pipeline run by using the locked version details to fetch exactly the same data. Monitoring systems can detect dataset drift by comparing current source state against locked state. And orchestration tools can determine whether a pipeline needs re-execution by checking if locked versions are still current.
Usage
Dataset resolution is invoked whenever:
- A pipeline consumer needs to programmatically retrieve the location and version of an input dataset.
- A reproducibility check compares the current state of a data source against its locked version.
- An API client calls
datasets_show()to obtain the full dataset definition for a given name and revision. - A data monitoring system inspects locked dataset metadata to detect staleness or drift.
- An orchestrator evaluates whether pipeline re-execution is necessary based on dataset version changes.
Theoretical Basis
Declaration-lock duality. Dataset resolution embodies a two-layer versioning pattern found broadly in dependency management systems. The declaration specifies intent ("I depend on this data source"), while the lock records a resolved snapshot ("at this exact version"). This pattern is directly analogous to the relationship between a package manifest (e.g., requirements.txt) and a lock file (e.g., requirements.lock) in software dependency management:
Declaration (dvc.yaml):
dataset "sales_data":
type: url
url: https://data.example.com/sales/latest.csv
Lock (dvc.lock):
dataset "sales_data":
etag: "a3f7b2c1..."
content_hash: "md5:9e107d9d..."
timestamp: "2025-12-15T08:30:00Z"
Resolved Dataset:
name: "sales_data"
type: url
url: https://data.example.com/sales/latest.csv
lock:
etag: "a3f7b2c1..."
content_hash: "md5:9e107d9d..."
timestamp: "2025-12-15T08:30:00Z"
Type-directed resolution. Different dataset types require different resolution strategies. The resolver uses the declared type to determine which fields to extract and how to interpret lock information. This follows the strategy pattern, where the resolution algorithm varies based on the dataset type while presenting a uniform interface to consumers:
function resolve_dataset(name, revision):
declaration = load_dvc_yaml(revision).datasets[name]
lock_info = load_dvc_lock(revision).datasets[name]
match declaration.type:
case "url":
return URLDataset(url=declaration.url, etag=lock_info.etag, ...)
case "storage":
return StorageDataset(bucket=declaration.bucket, key=declaration.key, hash=lock_info.hash, ...)
case "db":
return DBDataset(connection=declaration.connection, query=declaration.query, checksum=lock_info.checksum, ...)
This type-directed dispatch ensures that each dataset type's resolution produces a result object with the correct semantics, while the overall resolution API remains uniform and type-agnostic for consumers.