Implementation:Treeverse LakeFS Esti Test Utilities

Knowledge Sources	Treeverse_LakeFS
Domains	Testing, Integration Tests, Go
Last Updated	2026-02-08 00:00 GMT

Overview

The esti_utils.go file provides Go helper functions and types used across the lakeFS end-to-end integration test suite (known as "esti") for test setup, API operations, S3 operations, and verification.

Description

This 694-line Go source file in the esti package contains the core utility layer for lakeFS integration tests. It defines shared types, constants, and helper functions that abstract common operations such as creating and deleting repositories, uploading objects, listing resources, managing test environments, and verifying results.

Key exported types:

• ArrayFlags - A string slice type implementing the flag.Value interface for multi-value CLI flags
• HookResponse - A struct capturing webhook test responses with fields: Path (string), Err (error), Data ([]byte), and QueryParams (map[string][]string)

Key exported constants:

• DefaultAdminAccessKeyID - Default test admin AWS-style access key ID
• DefaultAdminSecretAccessKey - Default test admin AWS-style secret access key
• AdminUsername - The default admin username ("esti")
• ViperStorageNamespaceKey - Viper config key for storage namespace
• ViperBlockstoreType - Viper config key for blockstore type

Key exported functions:

• EnvCleanup - Deletes all repositories, groups, policies, and users except those in keep-lists
• DeleteAllRepositories - Paginated deletion of all repositories not in the keep-list
• DeleteAllGroups - Paginated deletion of all groups not in the keep-list
• DeleteAllUsers - Paginated deletion of all users not in the keep-list
• DeleteAllPolicies - Paginated deletion of all policies not in the keep-list
• CleanupUser - Removes a specific user created during testing
• VerifyResponse - Checks an HTTP response for error status codes (>= 400)
• MakeRepositoryName - Sanitizes a string into a valid repository name by replacing non-alphanumeric characters
• GenerateUniqueRepositoryName - Creates a unique repo name using xid
• GenerateUniqueStorageNamespace - Creates a unique storage namespace path
• DeleteRepositoryIfAskedTo - Conditionally deletes a repository based on Viper config
• UploadFileRandomDataAndReport - Uploads a file with random 16-byte content, returning checksum and content
• UploadFileAndReport - Uploads a file via API or direct storage access, returning a checksum
• UploadContent - Uploads content as a multipart form to the lakeFS API
• UploadContentWithMetadata - Uploads content with custom metadata headers
• UploadFileRandomData - Test helper wrapping UploadFileRandomDataAndReport with require assertions
• ListRepositoryObjects - Paginated listing of all objects in a repository ref
• RequireBlockstoreType - Skips a test if the blockstore type does not match required types
• GravelerIterator - Creates a Graveler SSTable iterator from raw byte data
• WaitForListRepositoryRunsLen - Polls with exponential backoff until action runs reach expected count
• ResponseWithTimeout - Waits for a webhook response with a timeout
• CheckFilesWereGarbageCollected - Verifies garbage collection by checking presigned URL accessibility

Usage

Import the esti package in integration test files. Use setupTest to create a test context with a unique repository, then call the various helper functions to manipulate lakeFS resources and verify outcomes. Call tearDownTest to clean up after each test.

Code Reference

Source Location

• Repository: Treeverse_LakeFS
• File: esti/esti_utils.go
• Lines: 1-694

Signature

type ArrayFlags []string

type HookResponse struct {
	Path        string
	Err         error
	Data        []byte
	QueryParams map[string][]string
}

func EnvCleanup(ctx context.Context, client apigen.ClientWithResponsesInterface, repositoriesToKeep, groupsToKeep, usersToKeep, policiesToKeep ArrayFlags) error
func DeleteAllRepositories(ctx context.Context, client apigen.ClientWithResponsesInterface, repositoriesToKeep ArrayFlags) error
func DeleteAllGroups(ctx context.Context, client apigen.ClientWithResponsesInterface, groupsToKeep ArrayFlags) error
func DeleteAllUsers(ctx context.Context, client apigen.ClientWithResponsesInterface, usersToKeep ArrayFlags) error
func DeleteAllPolicies(ctx context.Context, client apigen.ClientWithResponsesInterface, policiesToKeep ArrayFlags) error
func CleanupUser(t testing.TB, ctx context.Context, client apigen.ClientWithResponsesInterface, userName string)
func VerifyResponse(resp *http.Response, body []byte) error
func MakeRepositoryName(name string) string
func GenerateUniqueRepositoryName() string
func GenerateUniqueStorageNamespace(repoName string) string
func DeleteRepositoryIfAskedTo(ctx context.Context, repositoryName string)
func UploadFileRandomDataAndReport(ctx context.Context, repo, branch, objPath string, direct bool, clt apigen.ClientWithResponsesInterface) (checksum, content string, err error)
func UploadFileAndReport(ctx context.Context, repo, branch, objPath, objContent string, direct bool, clt apigen.ClientWithResponsesInterface) (checksum string, err error)
func UploadContent(ctx context.Context, repo, branch, objPath, objContent string, clt apigen.ClientWithResponsesInterface) (*apigen.UploadObjectResponse, error)
func UploadContentWithMetadata(ctx context.Context, client apigen.ClientWithResponsesInterface, repo, branch, path string, metadata map[string]string, contentType string, content io.Reader) (*apigen.UploadObjectResponse, error)
func UploadFileRandomData(ctx context.Context, t *testing.T, repo, branch, objPath string, clt apigen.ClientWithResponsesInterface) (checksum, content string)
func ListRepositoryObjects(ctx context.Context, t *testing.T, repository string, ref string, clt apigen.ClientWithResponsesInterface) []apigen.ObjectStats
func RequireBlockstoreType(t testing.TB, requiredTypes ...string)
func GravelerIterator(data []byte) (*sstable.Iterator, error)
func WaitForListRepositoryRunsLen(ctx context.Context, t *testing.T, repo, ref string, l int, clt apigen.ClientWithResponsesInterface) *apigen.ActionRunList
func ResponseWithTimeout(s *WebhookServer, timeout time.Duration) (*HookResponse, error)
func CheckFilesWereGarbageCollected(t *testing.T, expectedExisting map[string]bool, presignedURLs map[string]string)

Import

import (
	// This file is part of package esti; used internally by test files in the same package.
	// No external import path - co-located in esti/ directory.
)

I/O Contract

Inputs

Name	Type	Required	Description
ctx	context.Context	Yes	Go context for cancellation and deadline propagation
client	apigen.ClientWithResponsesInterface	Yes	Auto-generated lakeFS API client
t	testing.TB	Yes	Go test context for assertions and logging
repo	string	Yes	Repository name for API operations
branch	string	Yes	Branch name for upload and listing operations
objPath	string	Yes	Object path within the repository

Outputs

Name	Type	Description
error	error	Nil on success; descriptive error wrapping errNotVerified or errWrongStatusCode on failure
checksum	string	MD5/ETag checksum returned after object upload
[]apigen.ObjectStats	slice	List of object metadata from paginated listing
*apigen.ActionRunList	pointer	Action run results after polling

Usage Examples

// Setting up a test with a unique repository
func TestMyFeature(t *testing.T) {
	ctx, log, repo := setupTest(t)
	defer tearDownTest(repo)

	// Upload a random file
	checksum, content := UploadFileRandomData(ctx, t, repo, "main", "path/to/file.txt", nil)

	// List objects in the repository
	objects := ListRepositoryObjects(ctx, t, repo, "main", client)

	// Skip test if blockstore is not S3
	RequireBlockstoreType(t, "s3")
}

Related Pages

• Principle:Treeverse_LakeFS_Integration_Test_Utilities