Implementation:Huggingface Transformers Tests Fetcher
| Knowledge Sources | |
|---|---|
| Domains | CI_CD, Testing_Infrastructure |
| Last Updated | 2026-02-13 20:00 GMT |
Overview
Concrete tool for determining which tests to run for a PR or commit by analyzing file diffs and building an import dependency graph across the codebase.
Description
The tests_fetcher.py utility (1153 lines) operates in multiple stages: (1) Identifies modified Python files by diffing the current branch against main (or the last commit), filtering out docstring-only changes. (2) Builds a complete import dependency map by parsing relative and direct imports from every module and test file using regex patterns. (3) Creates a reverse dependency map so that for any modified file, all transitively dependent test files can be identified. (4) If too many models are impacted (15+) or core files are modified, triggers full CI. It also handles doctest file identification, supports commit message flags like [ci skip], [no filter], [test all], and creates per-job test lists matching regex patterns in JOB_TO_TEST_FILE. This is the primary mechanism for selective test execution in the CI pipeline.
Usage
Run at the start of CI to determine which test files need to be executed based on the current changes.
Code Reference
Source Location
- Repository: Huggingface_Transformers
- File: utils/tests_fetcher.py
- Lines: 1-1153
Signature
def get_diff(base: str = "main") -> List[str]:
"""Get list of modified files compared to base branch."""
def get_module_dependencies(file_path: str) -> Set[str]:
"""Parse imports from a Python file and return set of dependencies."""
def get_test_files_for_modified_files(
modified_files: List[str],
dependency_map: Dict[str, Set[str]],
) -> Set[str]:
"""Find all test files that transitively depend on modified files."""
def infer_tests_to_run(
base: str = "main",
output_dir: str = "test_preparation",
) -> None:
"""Main entry point: compute and write per-job test lists."""
Import
python utils/tests_fetcher.py
python utils/tests_fetcher.py --base main --output_dir test_preparation
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| Git diff | Git state | Yes | Current branch changes vs base |
| --base | str | No | Base branch for diff (default: main) |
| --output_dir | str | No | Directory for test list output files |
Outputs
| Name | Type | Description |
|---|---|---|
| *_test_list.txt | Text files | Per-job test file lists in output directory |
| test_repo_utils.txt | Text file | Test list for repo utility tests |
Usage Examples
Selective Test Execution
# Determine tests to run for current branch
python utils/tests_fetcher.py
# With custom base branch
python utils/tests_fetcher.py --base v4.35.0
# Output to custom directory
python utils/tests_fetcher.py --output_dir ./my_test_lists/