Implementation:Huggingface Transformers Notification Service
| Knowledge Sources | |
|---|---|
| Domains | CI_CD, Observability |
| Last Updated | 2026-02-13 20:00 GMT |
Overview
Concrete tool for collecting CI test results from GitHub Actions workflow artifacts and sending formatted Slack reports.
Description
The notification_service.py utility (1602 lines) is the primary CI reporting mechanism. It retrieves available test artifacts from the filesystem, processes each artifact to extract failure counts, stacktraces, and timing data. The Message class builds structured Slack block-kit payloads that categorize failures by model, GPU type (single/multi), and test category (PyTorch, Tokenizers, Pipelines, etc.). It compares current results against previous CI run artifacts (fetched from HuggingFace Hub) to compute new failures and diffs, uploads result JSON files to a Hub dataset repository, and posts the report to a Slack channel via slack_sdk.WebClient. Handles both Nvidia and AMD CI workflows and supports push, scheduled, and issue-comment triggered runs.
Usage
Run automatically at the end of CI workflow runs to post test result summaries to Slack.
Code Reference
Source Location
- Repository: Huggingface_Transformers
- File: utils/notification_service.py
- Lines: 1-1602
Signature
class Message:
"""Builds Slack block-kit message payloads for CI results."""
def __init__(
self,
title: str,
ci_title: str,
results: Dict,
prev_ci_artifacts: Dict = None,
):
"""Initialize with test results and optional previous run data."""
class Artifact:
"""Represents a CI test artifact with parsed results."""
def handle_test_results(artifact: Artifact) -> Dict:
"""Extract test counts, failures, and timing from an artifact."""
def prepare_reports(
artifact_dir: str,
job_name: str,
) -> Tuple[Dict, List[str]]:
"""Prepare structured reports from CI artifacts."""
Import
python utils/notification_service.py
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| Test artifacts | Directory | Yes | CI test result files from GitHub Actions |
| SLACK_TOKEN | env var | Yes | Slack API token for posting messages |
| GITHUB_TOKEN | env var | Yes | GitHub token for artifact access |
Outputs
| Name | Type | Description |
|---|---|---|
| Slack message | Slack post | Formatted test result summary in Slack channel |
| Hub uploads | JSON files | Test results uploaded to HuggingFace Hub dataset |
Usage Examples
CI Pipeline Usage
# Run in CI after all test jobs complete
python utils/notification_service.py \
--ci_event push \
--artifact_dir ./test_artifacts/