Implementation:Huggingface Transformers Notification Service Doc Tests
| Knowledge Sources | |
|---|---|
| Domains | CI_CD, Documentation |
| Last Updated | 2026-02-13 20:00 GMT |
Overview
Concrete tool for collecting doctest CI results from GitHub Actions artifacts and sending Slack notifications for documentation test failures.
Description
The notification_service_doc_tests.py utility is specialized for documentation tests. It retrieves artifacts prefixed with doc_tests_gpu_test_reports_, parses test statistics (passed/failed counts, timing), extracts first-line failure messages from doctest output, and builds a Message object. The Message class formats Slack block-kit payloads with failure summaries organized by category (Python Examples vs MD Examples). Results are saved to doc_test_results/doc_test_results.json and posted to Slack with threaded replies for each failing job.
Usage
Run automatically at the end of doctest CI workflow runs to post documentation test results to Slack.
Code Reference
Source Location
- Repository: Huggingface_Transformers
- File: utils/notification_service_doc_tests.py
- Lines: 1-384
Signature
class Message:
"""Builds Slack block-kit payloads for doctest results."""
class Artifact:
"""Represents a doctest CI artifact."""
def handle_test_results(artifact: Artifact) -> Dict:
"""Extract doctest pass/fail counts and failure messages."""
def extract_first_line_failure(failure_text: str) -> str:
"""Extract the first meaningful line from a doctest failure."""
Import
python utils/notification_service_doc_tests.py
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| doc_tests_gpu_test_reports_* | Artifacts | Yes | Doctest result artifacts |
| SLACK_TOKEN | env var | Yes | Slack API token |
Outputs
| Name | Type | Description |
|---|---|---|
| Slack message | Slack post | Doctest result summary in Slack |
| doc_test_results.json | JSON | Structured doctest results |
Usage Examples
CI Pipeline Usage
# Run after doctest jobs complete
python utils/notification_service_doc_tests.py