Implementation:Huggingface Transformers Check Bad Commit

Knowledge Sources	Huggingface_Transformers
Domains	CI_CD, Debugging
Last Updated	2026-02-13 20:00 GMT

Overview

Concrete tool for identifying the specific git commit that introduced a test failure using automated git bisect.

Description

The check_bad_commit.py utility automates the process of finding which commit broke a specific test. It generates a temporary Python script that installs the repository and runs a target test with pytest --flake-finder. It uses git bisect run to binary-search through commit history between a known good commit and a bad commit. When the culprit commit is found, it queries the GitHub API to find the associated pull request and its author. The is_bad_commit function can also verify a single commit in isolation.

Usage

Use this tool when a CI test starts failing and you need to pinpoint which commit caused the regression. Provide a known good commit SHA and a bad commit SHA along with the failing test path.

Code Reference

Source Location

• Repository: Huggingface_Transformers
• File: utils/check_bad_commit.py
• Lines: 1-312

Signature

def create_script(test: str) -> str:
    """Generate a temporary test runner script for git bisect."""

def is_bad_commit(commit_id: str, test: str) -> bool:
    """Check if a specific commit causes the given test to fail."""

def find_bad_commit(good: str, bad: str, test: str) -> str:
    """Use git bisect to find the first bad commit between good and bad."""

def get_commit_info(commit_id: str) -> Dict:
    """Query GitHub API for PR info associated with a commit."""

Import

python utils/check_bad_commit.py --good abc123 --bad def456 \
    --test tests/models/bert/test_modeling_bert.py::BertModelTest::test_forward

I/O Contract

Inputs

Name	Type	Required	Description
--good	str	Yes	Known good commit SHA
--bad	str	Yes	Known bad commit SHA
--test	str	Yes	Full pytest test path to check

Outputs

Name	Type	Description
Bad commit SHA	stdout	The commit that introduced the failure
PR info	stdout	Associated pull request number and author

Usage Examples

Finding the Bad Commit

# Find which commit broke a test
python utils/check_bad_commit.py \
    --good v4.35.0 \
    --bad main \
    --test tests/models/bert/test_modeling_bert.py::BertModelTest::test_forward

# Check if a specific commit is bad
python utils/check_bad_commit.py \
    --commit abc123 \
    --test tests/models/bert/test_modeling_bert.py::BertModelTest::test_forward

Related Pages

• Principle:Huggingface_Transformers_Commit_Bisection_Debugging