Principle:Treeverse LakeFS Integration Test Utilities
| Knowledge Sources | |
|---|---|
| Domains | Testing, Integration Tests, CLI, Go |
| Last Updated | 2026-02-08 00:00 GMT |
Overview
lakeFS maintains a dedicated integration test infrastructure (the "esti" suite) with shared utility libraries that abstract API and CLI interactions, enabling consistent, reproducible, and maintainable end-to-end tests.
Description
The lakeFS project follows the principle of investing in a robust, layered integration test utility layer that separates test logic from infrastructure concerns. The "esti" test suite (named after the default admin user) provides two complementary utility modules:
1. API-Level Test Utilities (esti_utils.go): These helpers interact directly with the lakeFS HTTP API through the auto-generated apigen client. They provide:
- Environment lifecycle management -- creating unique repositories with random names (via xid), setting up storage namespaces, and tearing down resources after tests
- CRUD operations -- uploading objects (via multipart form or direct storage access), listing repository contents with pagination, and managing branches
- Bulk cleanup -- deleting all repositories, users, groups, and policies except configurable keep-lists, enabling clean test environments
- Verification helpers -- checking HTTP response codes, waiting for action runs with exponential backoff, and validating garbage collection via presigned URLs
- Blockstore awareness -- skipping tests that require specific storage backends (S3, GCS, Azure) when running against a different blockstore
2. CLI-Level Test Utilities (lakectl_util.go): These helpers test the lakectl command-line interface through shell execution and output comparison. They provide:
- Golden file testing -- comparing sanitized CLI output against stored .golden reference files, with an -update flag to regenerate them
- Output sanitization -- normalizing non-deterministic values (timestamps, commit IDs, checksums, endpoint URLs, access keys, physical addresses) using regular expressions so that output comparisons are stable across runs
- Variable expansion/embedding -- a bidirectional variable system where ${VAR_NAME} placeholders in golden files are expanded with run-specific values, and run-specific values in output are embedded back as variables
- Command execution -- running lakectl with configurable credentials, endpoint URLs, and POSIX permission settings
Together, these utilities ensure that integration tests are:
- Isolated -- each test gets a unique repository and namespace
- Deterministic -- non-deterministic output is sanitized before comparison
- Self-cleaning -- resources are torn down after tests complete
- Maintainable -- common patterns are abstracted into reusable helpers
Usage
Apply this principle whenever:
- Writing new lakeFS integration tests -- use the existing helpers rather than making raw API calls
- Adding a new lakectl command -- create a golden file and use RunCmdAndVerifySuccessWithFile to validate output
- The API output format changes -- update golden files by running tests with the -update flag
- Extending the test infrastructure -- add new helpers to the appropriate utility file (API-level in esti_utils.go, CLI-level in lakectl_util.go)
- Debugging test failures -- check whether sanitization regex patterns need updating for new output formats
Theoretical Basis
The lakeFS integration test infrastructure embodies several well-established testing principles:
Test Utility Layer Pattern: Rather than duplicating setup and teardown code across dozens of test files, common operations are centralized in utility modules. This reduces code duplication, makes tests more readable, and ensures consistent behavior (e.g., all repository names are sanitized the same way, all uploads use the same multipart format).
Golden File Testing: The lakectl test utilities use the golden file pattern (also known as "snapshot testing"), where expected output is stored in versioned reference files. This is particularly effective for CLI tools where output format matters. The bidirectional variable system (expand/embed) makes golden files portable across different test runs by abstracting away run-specific values like repository names and commit IDs.
Output Normalization: Non-deterministic values (timestamps, UUIDs, checksums) are replaced with stable placeholders before comparison. This ensures tests do not produce false failures due to inherently variable data. The normalization is applied in a specific order to handle cases where one pattern might contain another (e.g., endpoint URLs within pre-signed URLs).
Environment Isolation: Each test creates its own uniquely-named repository using xid for collision-free names. The EnvCleanup function provides comprehensive teardown by paginating through all resources and deleting those not in a configurable keep-list, preventing test pollution.
Exponential Backoff Polling: The WaitForListRepositoryRunsLen function uses exponential backoff (via the backoff library) to poll for eventually-consistent results, avoiding both premature failures and unnecessary delays in asynchronous operations like action hook execution.