Principle:Explodinggradients Ragas Experiment Versioning
| Knowledge Sources | Domains | Last Updated |
|---|---|---|
| explodinggradients/ragas | Experiment Management, Reproducibility | 2026-02-10 |
Overview
Experiment Versioning is the principle of using Git-based version control to snapshot the codebase state for each evaluation experiment, enabling reproducibility and comparison across iterations.
Description
Evaluation experiments in LLM applications are highly sensitive to code changes: a modified prompt template, an updated metric threshold, or a changed data preprocessing step can significantly alter results. Experiment Versioning addresses this by creating Git snapshots that associate each experiment run with the exact codebase state that produced it.
Git-Based Snapshots: Rather than inventing a custom versioning system, the principle leverages Git, the most widely used version control system. Each experiment version is captured as a Git commit, providing a complete, auditable record of every file in the project at the time of the experiment.
Branch-Per-Experiment: Optionally, each experiment creates a named Git branch (using a ragas/{experiment_name} convention). This makes it easy to navigate between experiment versions using standard Git tools. The branch points to the commit that captured the experiment state, serving as a human-readable bookmark.
Selective Staging: The versioning mechanism supports two staging modes. By default, only tracked files with modifications are staged (git add -u), which is the safe default that avoids accidentally committing untracked files like data dumps or credentials. An optional stage_all mode stages everything including untracked files (git add .), useful when new files are integral to the experiment.
Idempotent Behavior: If no changes are detected in the repository, the versioning function does not create an empty commit. Instead, it returns the current HEAD commit hash, avoiding clutter in the Git history while still providing a reference point.
Reproducibility Chain: The returned commit hash serves as a reproducibility key. Combined with the experiment name stored in the backend, this creates a chain linking experiment results to the exact code that produced them.
Usage
Use the Experiment Versioning principle when:
- Running evaluation experiments that must be reproducible
- Comparing results across multiple iterations of evaluation code
- Needing to audit which code version produced specific evaluation results
- Creating experiment branches for team collaboration on evaluation development
- Building CI/CD pipelines for automated evaluation with version tracking
Theoretical Basis
The theoretical foundation applies version control theory to the experiment lifecycle:
PROCEDURE version_experiment(experiment_name, commit_message, repo_path, create_branch, stage_all):
1. Locate the Git repository:
IF repo_path is not provided:
Search upward from the current directory for a .git directory
Initialize a Git Repo object at the resolved path
2. Check for and stage changes:
IF stage_all AND repo has any changes (including untracked):
Stage all files: git add .
Mark has_changes = True
ELIF repo has changes to tracked files only:
Stage tracked file changes: git add -u
Mark has_changes = True
3. Create the version commit:
IF has_changes:
Use provided commit_message OR default to "Experiment: {experiment_name}"
Create commit with the staged changes
Record the commit hash
ELSE:
Use the current HEAD commit hash (no empty commit)
4. Create the experiment branch:
IF create_branch:
Create branch named "ragas/{experiment_name}" pointing to the commit
5. Return the commit hash as the version identifier
This approach ensures every experiment has a traceable link to the code that produced it, using standard Git primitives that integrate with existing development workflows.