Implementation:Huggingface Transformers Get CI Error Statistics
| Knowledge Sources | |
|---|---|
| Domains | CI_CD, Observability |
| Last Updated | 2026-02-13 20:00 GMT |
Overview
Concrete tool for collecting, aggregating, and summarizing test failure statistics from GitHub Actions CI workflow runs.
Description
The get_ci_error_statistics.py utility uses the GitHub API to fetch job links and artifact download URLs from a workflow run (handling pagination for runs with 100+ jobs). It downloads each artifact zip file, then extracts error information from failures_line.txt, summary_short.txt, and job_name.txt files within each artifact. It correlates error lines with failed test names and job links. Aggregates errors using Counter into two views: reduce_by_error (counts each unique error across all tests) and reduce_by_model (counts errors per model directory). Outputs results as GitHub-formatted markdown tables and JSON files. Also serves as a shared library imported by notification_service.py and extract_warnings.py.
Usage
Run standalone to analyze a specific CI workflow run, or used as a library by the notification service.
Code Reference
Source Location
- Repository: Huggingface_Transformers
- File: utils/get_ci_error_statistics.py
- Lines: 1-310
Signature
def get_jobs(workflow_run_id: str, token: str) -> List[Dict]:
"""Fetch all jobs for a GitHub Actions workflow run."""
def get_artifacts_links(workflow_run_id: str, token: str) -> Dict[str, str]:
"""Get download URLs for all artifacts in a workflow run."""
def download_artifact(url: str, token: str, output_dir: str) -> str:
"""Download and extract a workflow artifact zip file."""
def reduce_by_error(errors: List[Dict]) -> Dict[str, int]:
"""Aggregate errors by unique error message."""
def reduce_by_model(errors: List[Dict]) -> Dict[str, int]:
"""Aggregate errors by model directory."""
Import
python utils/get_ci_error_statistics.py --workflow_run_id 12345 --token $GH_TOKEN
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| --workflow_run_id | str | Yes | GitHub Actions workflow run ID |
| --token | str | Yes | GitHub API token |
Outputs
| Name | Type | Description |
|---|---|---|
| Error tables | Markdown | GitHub-formatted tables of errors by type and by model |
| errors.json | JSON | Structured error data |
Usage Examples
Analyzing CI Failures
# Analyze a specific workflow run
python utils/get_ci_error_statistics.py \
--workflow_run_id 12345678 \
--token $GITHUB_TOKEN