Implementation:Huggingface Transformers Create Circleci Config

Knowledge Sources	Huggingface_Transformers
Domains	CI_CD, DevOps
Last Updated	2026-02-13 20:00 GMT

Overview

Concrete tool for dynamically generating CircleCI continuation configuration YAML based on selective test execution results.

Description

The create_circleci_config module generates a CircleCI continuation config by reading test lists produced by the tests_fetcher and mapping them to predefined CI job configurations. It defines a CircleCIJob dataclass that encapsulates job parameters (Docker image, pytest options, markers, parallelism, resource class, environment variables, install steps). Pre-configured job instances exist for torch, generate, tokenization, processors, pipelines, custom tokenizers, examples, hub, exotic models, repo utils, non-model tests, training CI, and doc tests. The main function scans a test_preparation folder for *_test_list.txt files, determines which jobs have tests to run, builds a complete CircleCI config dict with parameters, jobs, and workflows, and writes it as generated_config.yml.

Usage

Run this script as part of the CircleCI dynamic configuration pipeline to generate a continuation config that only includes jobs for tests affected by the current changes. It is invoked automatically by the CI system after the tests_fetcher determines which tests to run.

Code Reference

Source Location

• Repository: Huggingface_Transformers
• File: .circleci/create_circleci_config.py
• Lines: 1-413

Signature

@dataclass
class CircleCIJob:
    name: str
    additional_env: Dict[str, str] = None
    docker_image: List[Dict[str, str]] = None
    install_steps: List[str] = None
    marker: str = ""
    parallelism: int = 1
    pytest_num_workers: int = 8
    pytest_options: Dict[str, str] = None
    resource_class: str = "2xlarge"
    tests_to_run: Optional[List[str]] = None
    working_directory: str = "~/transformers"

def create_circleci_config(folder: str = None) -> None:
    """Generate CircleCI continuation config from test lists."""

Import

python .circleci/create_circleci_config.py

I/O Contract

Inputs

Name	Type	Required	Description
test_preparation folder	Directory	Yes	Directory containing *_test_list.txt files from tests_fetcher
--folder	str	No	Override the test preparation folder path

Outputs

Name	Type	Description
generated_config.yml	YAML file	Complete CircleCI continuation configuration

Usage Examples

CI Pipeline Usage

# Typically called automatically by CircleCI dynamic config
python .circleci/create_circleci_config.py

# Or with a custom test preparation folder
python .circleci/create_circleci_config.py --folder /path/to/test_preparation

Related Pages

• Principle:Huggingface_Transformers_CI_Configuration_Generation