Implementation:FMInference FlexLLMGen Transformers Setup

Field	Value
Sources	Repo: FlexLLMGen, Upstream: HuggingFace Transformers
Domains	Build_System, Package_Distribution
Last Updated	2026-02-09 00:00 GMT

Overview

Vendored HuggingFace Transformers package setup script that configures an extensive dependency table, custom build commands, and package metadata for producing distributable Python packages.

Description

The setup.py file (443 lines) is a vendored copy of HuggingFace Transformers' package build configuration, included in FlexLLMGen's benchmark third-party dependencies. It manages a large dependency ecosystem and provides specialized build commands.

Key components include:

• Dependency management:
  • _deps -- A comprehensive list of ~80+ pinned and range-specified dependencies covering machine learning frameworks (PyTorch, TensorFlow, JAX, Flax), tokenizers, datasets, evaluation tools, serving infrastructure, and language-specific NLP tools.
  • deps -- A lookup table mapping package names to their versioned specifiers, generated from _deps and stored in src/transformers/dependency_versions_table.py for runtime access.
  • deps_list() -- Helper function that resolves package names to versioned specifiers from the lookup table.

• Extras organization:
  • dev -- Development dependencies (testing, linting, documentation)
  • torch / tf / flax -- Framework-specific dependencies
  • serving -- FastAPI, Starlette, uvicorn for model serving
  • audio / vision -- Modality-specific dependencies (librosa, Pillow, timm)
  • quality / testing -- Code quality and test framework dependencies
  • Many other specialized groups for specific model families and features.

• Custom build commands:
  • DepsTableUpdateCommand -- A custom distutils command (python setup.py deps_table_update) that regenerates the dependency versions table from the _deps list, ensuring runtime and build-time dependency specifications stay in sync.

• Stale egg-info cleanup:
  • Automatically removes stale transformers.egg-info directories that can cause pip installation issues (workaround for pip issue #5466).

• Package discovery:
  • Uses find_packages() with src as the package root directory.

Usage

This setup script is invoked during pip install of the vendored Transformers package in FlexLLMGen's benchmark suite. The vendored copy may contain modifications specific to FlexLLMGen's benchmarking needs.

Code Reference

Field	Value
Repository	FlexLLMGen
File	benchmark/third_party/transformers/setup.py
Lines	1-443
Type	AUTO_KEEP (vendored dependency)

Key dependency examples:

_deps = [
    "tokenizers>=0.11.1,!=0.11.3,<0.14",
    "torch>=1.7,!=1.12.0",
    "huggingface-hub>=0.10.0,<1.0",
    "numpy>=1.17",
    "packaging>=20.0",
    "tqdm>=4.27",
    "safetensors>=0.2.1",
    "deepspeed>=0.6.5",
    # ... 80+ total dependencies
]

deps = {b: a for a, b in (re.findall(r"^(([^!=<>~ ]+)(?:[!=<>~ ].*)?$)", x)[0] for x in _deps)}

I/O Contract

Inputs

Parameter	Type	Required	Description
_deps	list[str]	Internal	Master list of all dependencies with version constraints
extras groups	dict	Internal	Groups of optional dependencies by feature area

Outputs

Output	Type	Description
transformers package	wheel/sdist	Installable Python package
dependency_versions_table.py	Python module	Runtime-accessible dependency version specifications

Related Pages

• Principle:FMInference_FlexLLMGen_Transformers_Package_Build
• Implementation:FMInference_FlexLLMGen_DeepSpeed_Setup