Implementation:Microsoft LoRA Check Repo

Template:Implementation meta

Overview

The check_repo.py utility validates the HuggingFace Transformers repository by ensuring all models are properly tested, documented, and registered in auto-mapping classes.

Description

This CI validation script performs three comprehensive quality checks on the Transformers repository:

1. Model Testing Coverage: Iterates over all modeling modules under src/transformers/models/, finds the corresponding test files, and verifies that every model class appears in all_model_classes within its test file. Models listed in IGNORE_NON_TESTED (e.g., encoder/decoder sub-components like BartEncoder, BartDecoderWrapper) are exempted.

1. Documentation Coverage: Parses all RST documentation files to extract documented classes/functions, then compares against all public objects in the Transformers init. Exemptions are managed through DEPRECATED_OBJECTS, UNDOCUMENTED_OBJECTS, and rules in ignore_undocumented() (e.g., constants, PreTrainedModel subclasses, internal modules).

1. Auto-Configuration Coverage: Checks that every model class appears in at least one MODEL_*_MAPPING or TF_MODEL_*_MAPPING auto class. Models in IGNORE_NON_AUTO_CONFIGURED are exempted.

Additionally, it validates decorator ordering in test files, ensuring that @parameterized decorators always appear first when stacked with other decorators.

Usage

Use this utility when:

• Running CI quality gates to catch untested or undocumented models.
• Adding a new model to verify it is properly integrated (tested, documented, auto-configured).
• Diagnosing test coverage gaps for specific model architectures.

Code Reference

Source Location

examples/NLU/utils/check_repo.py (501 lines)

Signature

def get_model_modules() -> list: ...
def get_models(module) -> list: ...
def get_model_test_files() -> list: ...
def find_tested_models(test_file: str) -> list: ...
def check_models_are_tested(module, test_file: str) -> list: ...
def check_all_models_are_tested() -> None: ...
def get_all_auto_configured_models() -> list: ...
def check_models_are_auto_configured(module, all_auto_models: list) -> list: ...
def check_all_models_are_auto_configured() -> None: ...
def check_decorator_order(filename: str) -> list: ...
def check_all_decorator_order() -> None: ...
def find_all_documented_objects() -> list: ...
def ignore_undocumented(name: str) -> bool: ...
def check_all_objects_are_documented() -> None: ...
def check_repo_quality() -> None: ...

Import / CLI Usage

# Run from repository root
python utils/check_repo.py

I/O Contract

Inputs

Input	Type	Description
src/transformers/	Directory	Source modules dynamically imported to discover model classes
tests/	Directory	Test files scanned for all_model_classes declarations
docs/source/**/*.rst	Files	RST documentation files parsed for autoclass and autofunction directives

Outputs

Output	Type	Description
Exception	Exception	Raised with a summary of all failures (untested models, undocumented objects, missing auto-configurations)
Console output	stdout	Progress messages for each check phase

Usage Examples

# Run all repository quality checks
python utils/check_repo.py

# Output on success:
# Checking all models are properly tested.
# Checking all objects are properly documented.
# Checking all models are in at least one auto class.

# On failure (example):
# Exception: There were 2 failures:
# - BertForNewTask is defined in transformers.models.bert.modeling_bert but is not tested in tests/test_modeling_bert.py
# - BertForNewTask is defined in transformers.models.bert.modeling_bert but is not present in any of the auto mapping

Related Pages

• Environment:Microsoft_LoRA_NLU_Conda_Environment