Implementation:Huggingface Transformers Update Metadata
| Knowledge Sources | |
|---|---|
| Domains | Repository_Maintenance, Hub_Integration |
| Last Updated | 2026-02-13 20:00 GMT |
Overview
Concrete tool for updating the huggingface/transformers-metadata Hub dataset with current model framework support, pipeline tags, and auto-class mappings.
Description
The update_metadata.py utility keeps Hub metadata in sync with the library. It builds a frameworks table by scanning all transformers model classes to determine which model types have PyTorch implementations. It determines the appropriate processor class (AutoProcessor, AutoTokenizer, AutoImageProcessor, or AutoFeatureExtractor) for each model type. Updates a pipeline tags table by mapping model classes to pipeline tags (e.g., text-generation, image-classification) and auto-model classes using the PIPELINE_TAGS_AND_AUTO_MODELS constant. Compares generated JSON against current Hub versions and only uploads if changes are detected.
Usage
Run periodically or after adding new models to keep the Hub metadata repository in sync with the library.
Code Reference
Source Location
- Repository: Huggingface_Transformers
- File: utils/update_metadata.py
- Lines: 1-348
Signature
def get_frameworks_table() -> Dict[str, Dict[str, bool]]:
"""Build table of which frameworks each model type supports."""
def update_pipeline_and_auto_class_table() -> Dict:
"""Build mapping of model types to pipeline tags and auto classes."""
def update_metadata(
token: str = None,
check_only: bool = False,
) -> None:
"""Update Hub metadata dataset, or check for needed updates."""
def check_pipeline_tags() -> List[str]:
"""Validate all pipeline tasks have metadata entries."""
Import
python utils/update_metadata.py
python utils/update_metadata.py --check-only
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| src/transformers/ | Directory | Yes | Source tree for model introspection |
| --token | str | No | HuggingFace Hub API token for uploads |
| --check-only | flag | No | Validate without uploading |
Outputs
| Name | Type | Description |
|---|---|---|
| Hub dataset update | Hub API | Updated metadata files on huggingface/transformers-metadata |
| Validation report | stdout | Missing pipeline tag entries (if --check-only) |
Usage Examples
Updating Hub Metadata
# Update metadata on the Hub
python utils/update_metadata.py --token $HF_TOKEN
# Check for needed updates without uploading
python utils/update_metadata.py --check-only