Principle:Lance format Lance Version Listing
| Knowledge Sources | |
|---|---|
| Domains | Data_Engineering, Version_Control |
| Last Updated | 2026-02-08 19:00 GMT |
Overview
Version listing is the capability to enumerate all historical versions of a Lance dataset, providing metadata about each version for inspection and navigation.
Description
Every Lance dataset maintains a sequence of manifest files, one per version. Version listing reads all available manifests and returns a sorted collection of Version structs, each carrying the version number, creation timestamp, and user-supplied metadata. This enables users to audit dataset history, compare versions, and select a specific point in time for checkout or restoration.
The Version struct is intentionally lightweight. It exposes:
- version -- the monotonically increasing
u64identifier. - timestamp -- a UTC
DateTimerecording when the version was created. - metadata -- a
BTreeMap<String, String>containing summary information such as the operation type, number of fragments modified, and any user-provided tags.
Internally, version listing is driven by the CommitHandler::list_manifest_locations() method, which streams manifest paths from the object store. Each manifest is then deserialized to extract the version metadata. Results are sorted by version number in ascending order before being returned.
Usage
Use version listing to:
- Audit data lineage -- understand what changes occurred and when.
- Select a version for checkout -- browse versions to find the right one before calling
checkout_version(). - Monitor ingestion pipelines -- track how many versions have been produced and their timestamps.
- Implement garbage-collection policies -- identify old versions that can be cleaned up.
Theoretical Basis
Manifest-based Version Discovery
Lance stores manifests under a well-known directory structure within the dataset base path. Two naming schemes exist:
- V1: Manifests are stored as
{base}/_versions/{version}.manifest. - V2: Manifests use zero-padded names that sort lexicographically in descending order, enabling efficient "latest version" lookups on lexically ordered object stores.
When listing, the commit handler streams all manifest locations. If the object store supports lexicographically ordered listing and the V2 naming scheme is in use, the listing can be performed in a single pass without in-memory sorting. Otherwise, all manifest paths are collected and sorted in memory. On systems where this fallback is needed (local filesystem, S3 Express), the I/O is typically fast enough that the in-memory sort is negligible.
Each manifest is a self-contained snapshot of the dataset state: schema, fragment list, index references, and metadata. The Version struct is derived from the manifest by extracting only the fields relevant for version browsing.