Workflow:Treeverse LakeFS External Data Import
| Knowledge Sources | |
|---|---|
| Domains | Data_Engineering, Data_Migration, Data_Lake_Management |
| Last Updated | 2026-02-08 10:00 GMT |
Overview
End-to-end process for importing existing data from external object storage (S3, Google Cloud Storage, Azure Blob Storage) into a lakeFS repository without copying the underlying data.
Description
This workflow describes how to bring existing data that resides in object storage under lakeFS version control. The import operation creates lakeFS metadata entries that reference the existing objects in-place, avoiding expensive data copies. Imports can target entire prefixes (directory-level) or individual objects, and multiple sources can be combined in a single import operation. The process is asynchronous, allowing monitoring of progress for large-scale imports.
Usage
Execute this workflow when you have existing data in object storage that you want to bring under lakeFS version control. Common triggers include: onboarding an existing data lake to lakeFS, ingesting data from external data producers, periodic bulk data loading from partner systems, or migrating data between storage namespaces while maintaining version history.
Execution Steps
Step 1: Prepare Import Sources
Identify the external object storage locations to import from. Each import source is defined by a storage URI (e.g., S3 path, GCS path, or Azure Blob path), a destination path within the lakeFS repository, and an import type (prefix for directories, object for individual files).
Key considerations:
- Source paths must be accessible from the lakeFS server
- You cannot import from the repository's own storage namespace
- Multiple source locations can be combined in a single import operation
- Each source specifies whether it is a prefix (directory) or individual object
Step 2: Initiate Import
Submit the import request to lakeFS, targeting a specific branch. The import operation is asynchronous — it returns an import ID immediately and processes the data in the background. A commit message and optional metadata are specified at this stage.
Key considerations:
- Imports always target a specific branch
- The operation returns an import ID for tracking
- A commit message must be provided and will be attached to the resulting commit
- Custom metadata can be attached to the import commit
Step 3: Monitor Import Progress
Poll the import status endpoint using the import ID to track progress. The status response includes the current state (in-progress, completed, or failed), progress metrics, and any error information.
Key considerations:
- Polling interval should be appropriate for the import size
- Import operations can be cancelled while in progress
- Failed imports do not create partial commits
- Large imports may run for extended periods depending on object count
Step 4: Verify Imported Data
After the import completes, verify that the expected objects are accessible on the target branch. List objects under the imported prefix to confirm completeness, and check object metadata to ensure integrity.
Key considerations:
- The import creates a commit on the target branch
- Imported objects are immediately available for read operations
- Object metadata (size, content type) is preserved from the source
- No physical data copy occurs — lakeFS references the original storage locations
Step 5: Commit and Tag
The import operation automatically creates a commit on the target branch. Optionally, create a tag to mark this import point for easy reference. This enables tracking exactly which data was imported and when.
Key considerations:
- The import commit captures the exact state of imported data
- Tags provide stable references for audit and reproducibility
- Import metadata (source paths, timestamps) is stored in the commit