Principle:Treeverse LakeFS Import Tagging
| Knowledge Sources | |
|---|---|
| Domains | Data_Import, Data_Engineering |
| Last Updated | 2026-02-08 00:00 GMT |
Overview
Import tagging is the practice of creating immutable, human-readable references (tags) to mark the exact data state produced by an import operation, enabling traceability, rollback, and audit capabilities.
Description
In a versioned data lake, every import operation produces a commit. While commits are identified by opaque hash strings (e.g., a1b2c3d4e5f6...), these identifiers are not meaningful to humans. Import tagging addresses this by assigning descriptive, immutable names to import commits.
A tag in lakeFS is a named pointer to a specific commit. Once created, a tag is immutable -- it always resolves to the same commit, even as the branch advances with subsequent operations. This immutability provides critical guarantees:
- Reproducibility -- Any query or pipeline referencing the tag will always see the same data, regardless of when it runs
- Rollback capability -- If a subsequent import introduces bad data, the previous import's tag provides a known-good state to revert to
- Audit trail -- Tags with descriptive names (e.g.,
import-2024-01-15,import-v1.0,pre-import-baseline) create a human-readable history of when data was imported and what state the repository was in at each milestone - Downstream coordination -- Other teams and systems can reference tagged states by name rather than commit hashes, simplifying cross-team communication
Import tagging is typically performed immediately after a successful import and verification. It is the final step in the import workflow, sealing the imported data state with a persistent name.
Usage
Use import tagging when:
- After every import -- As a standard operational practice, tag each import commit with a descriptive name incorporating a version number, date, or source identifier
- Before risky operations -- Tag the current state before running a new import, so you have a rollback point if the new import is incorrect
- Milestone marking -- When a specific import represents a significant data milestone (e.g., initial data load, quarterly refresh, schema migration)
- Cross-team handoffs -- When you need to communicate a specific data state to another team or system (e.g., "the ML training data is at tag
training-data-v3") - Compliance requirements -- When regulatory or governance policies require immutable references to specific data states
Theoretical Basis
Import tagging is an application of the broader version labeling pattern from version control systems. In Git, tags serve the same purpose: they provide human-readable names for specific commits in a repository's history.
TAG SEMANTICS:
tag_name --> commit_id (immutable mapping)
"import-v1.0" --> "a1b2c3d4..."
"import-2024-01-15" --> "e5f6a7b8..."
"pre-import-baseline" --> "c9d0e1f2..."
The immutability property distinguishes tags from branches:
BRANCH (mutable):
"main" --> commit_X (today)
"main" --> commit_Y (tomorrow, after new operations)
TAG (immutable):
"import-v1.0" --> commit_X (forever)
In the context of the import workflow, tagging fits into the final phase:
IMPORT WORKFLOW WITH TAGGING:
1. PREPARE: Define import locations (sources, destinations)
2. INITIATE: Start the async import job
3. MONITOR: Poll for completion
4. VERIFY: Confirm imported objects are correct
5. TAG: Create an immutable reference to the import commit
tag_name = "import-" + date_or_version
tag_ref = import_status.commit.id
Tag naming conventions are an important design decision. Common patterns include:
| Pattern | Example | Use Case |
|---|---|---|
| Date-based | import-2024-01-15 |
Daily or periodic imports |
| Version-based | import-v1.0, import-v1.1 |
Versioned dataset releases |
| Source-based | import-s3-production |
Identifying the data source |
| Composite | import-sales-2024-Q1-v2 |
Combining dataset name, period, and version |
| Pre/post markers | pre-import-2024-01-15 |
Marking state before an import for rollback |
The force parameter on tag creation controls whether an existing tag can be overwritten. In most import workflows, force should be false (the default) to preserve the immutability guarantee. Setting force: true is appropriate only when deliberately replacing a tag, such as updating a "latest-import" convenience pointer.