Principle:Treeverse LakeFS GC Job Execution
| Knowledge Sources | |
|---|---|
| Domains | Storage_Management, Data_Lifecycle |
| Last Updated | 2026-02-08 00:00 GMT |
Overview
GC job execution uses Apache Spark to perform distributed, scalable deletion of unreferenced objects from the underlying object storage based on metadata prepared by the lakeFS server.
Description
Garbage collection of unreferenced objects in a data lake can involve millions or even billions of objects spread across distributed storage systems (S3, GCS, Azure Blob Storage). Deleting these objects sequentially from the lakeFS server would be prohibitively slow and would place excessive load on a single process.
lakeFS addresses this by delegating the actual deletion to an Apache Spark job that runs outside the lakeFS server as a separate, independently scalable process. The Spark job:
- Reads the prepared metadata (commits CSV, addresses Parquet) generated by the
prepareGarbageCollectionCommitsAPI - Partitions the deletion work across Spark executors for parallel processing
- Issues bulk delete requests to the underlying object storage
- Reports results upon completion
The Spark job is distributed as a JAR file (lakefs-spark-client) and is typically executed inside a Docker container based on the treeverse/bitnami-spark image. It connects to the lakeFS API using access credentials provided via Spark configuration properties.
Usage
Execute the Spark GC job when:
- The metadata preparation phase has completed successfully and returned a
run_id - Sufficient compute resources are available to run a Spark job (either a local Spark cluster, EMR, Dataproc, or similar)
- The operator has verified (optionally) that the prepared metadata looks correct
Theoretical Basis
The architecture of the GC execution phase embodies the separation of concerns principle applied to distributed systems:
Control plane (lakeFS server): Maintains the commit graph, evaluates retention rules, and produces metadata about what to delete. This is a metadata-intensive, graph-traversal workload best suited to a single coordinated process.
Data plane (Spark job): Performs the actual I/O-intensive deletion operations against the object store. This is an embarrassingly parallel workload best suited to distributed execution.
This separation mirrors the architecture of many large-scale storage systems:
| System | Control Plane | Data Plane |
|---|---|---|
| lakeFS GC | lakeFS server (metadata preparation) | Spark job (object deletion) |
| HDFS | NameNode (metadata management) | DataNodes (block storage/deletion) |
| Kubernetes | API Server / etcd (desired state) | Kubelet (container lifecycle) |
The use of Docker containers for Spark execution provides:
- Reproducibility: The exact same Spark + JAR version runs in every environment
- Isolation: The GC job does not interfere with other Spark workloads
- Portability: The same container image works on local machines, CI/CD pipelines, and cloud-managed Spark services
The run ID serves as a correlation identifier that links the preparation phase to the execution phase, enabling:
- Idempotent retries: If the Spark job fails, it can be re-run with the same run ID
- Audit trails: Operators can trace which preparation run drove which deletion
- Concurrency safety: Multiple preparation runs do not interfere with each other