Implementation:Hiyouga LLaMA Factory Constants

Knowledge Sources	Hiyouga_LLaMA_Factory
Domains	Configuration, Model Registry
Last Updated	2026-02-06 19:00 GMT

Overview

Concrete central registry of constants, enums, and supported model definitions for the entire LLaMA Factory framework.

Description

This module serves as the configuration hub for LLaMA Factory, defining project-wide constants, enumeration types, and the comprehensive model support registry. It is structured into several sections:

Constants:

• IGNORE_INDEX = -100 -- Label index ignored during loss computation
• IMAGE_PLACEHOLDER, VIDEO_PLACEHOLDER, AUDIO_PLACEHOLDER -- Multimodal placeholders (configurable via environment variables)
• CHECKPOINT_NAMES -- Set of recognized model checkpoint filenames
• FILEEXT2TYPE -- File extension to dataset type mapping (arrow, csv, json, jsonl, parquet, txt)
• LAYERNORM_NAMES -- Names used to identify LayerNorm modules
• TRAINING_STAGES -- Mapping from human-readable stage names to codes (sft, rm, ppo, dpo, kto, pt)
• METHODS -- Supported fine-tuning methods (full, freeze, lora, oft)
• MCA_SUPPORTED_MODELS, MOD_SUPPORTED_MODELS, SUPPORTED_CLASS_FOR_S2ATTN -- Model compatibility sets

Enum Classes (all StrEnum):

• AttentionFunction -- auto, disabled, sdpa, fa2, fa3
• EngineName -- huggingface, vllm, sglang, ktransformers
• DownloadSource -- hf, ms (ModelScope), om (OpenMind)
• QuantizationMethod -- bnb, gptq, awq, aqlm, quanto, eetq, hqq, mxfp4, fp8
• RopeScaling -- linear, dynamic, yarn, llama3

Model Registry: The register_model_group function populates SUPPORTED_MODELS (OrderedDict mapping names to download sources) and DEFAULT_TEMPLATE (defaultdict mapping model names to template names). Hundreds of models are registered across families including Aya, BLOOM, ChatGLM, CodeGemma, DeepSeek, Falcon, Gemma, InternLM, LLaMA, Mistral, Phi, Qwen, StarCoder, Yi, and many more.

Usage

This module is imported throughout the entire codebase. Constants like IGNORE_INDEX and IMAGE_PLACEHOLDER are used in data processing, the model registry is used by the web UI for model selection, and the enums are used for configuration validation.

Code Reference

Source Location

• Repository: Hiyouga_LLaMA_Factory
• File: src/llamafactory/extras/constants.py
• Lines: 1-3429

Signature

# Key constants
IGNORE_INDEX: int = -100
IMAGE_PLACEHOLDER: str = os.getenv("IMAGE_PLACEHOLDER", "<image>")
VIDEO_PLACEHOLDER: str = os.getenv("VIDEO_PLACEHOLDER", "<video>")
AUDIO_PLACEHOLDER: str = os.getenv("AUDIO_PLACEHOLDER", "<audio>")
FILEEXT2TYPE: dict[str, str]
SUPPORTED_MODELS: OrderedDict[str, dict[DownloadSource, str]]
DEFAULT_TEMPLATE: defaultdict[str, str]

# Enum classes
class AttentionFunction(StrEnum): ...
class EngineName(StrEnum): ...
class DownloadSource(StrEnum): ...
class QuantizationMethod(StrEnum): ...
class RopeScaling(StrEnum): ...

# Registration function
def register_model_group(
    models: dict[str, dict[DownloadSource, str]],
    template: str | None = None,
    multimodal: bool = False,
) -> None: ...

Import

from llamafactory.extras.constants import (
    IGNORE_INDEX,
    IMAGE_PLACEHOLDER,
    VIDEO_PLACEHOLDER,
    AUDIO_PLACEHOLDER,
    SUPPORTED_MODELS,
    DEFAULT_TEMPLATE,
    EngineName,
    DownloadSource,
    QuantizationMethod,
)

I/O Contract

Inputs (register_model_group)

Name	Type	Required	Description
models	dict[str, dict[DownloadSource, str]]	Yes	Mapping of model display names to download source paths
template	str	No	Template name for chat/instruct variants
multimodal	bool	No	Whether the model supports multimodal inputs (default False)

Outputs (register_model_group)

Name	Type	Description
SUPPORTED_MODELS	OrderedDict	Updated global registry of all supported models
DEFAULT_TEMPLATE	defaultdict	Updated mapping from model names to default template names
MULTIMODAL_SUPPORTED_MODELS	set	Updated set of multimodal-capable model names

Usage Examples

from llamafactory.extras.constants import (
    IGNORE_INDEX,
    IMAGE_PLACEHOLDER,
    SUPPORTED_MODELS,
    DEFAULT_TEMPLATE,
    DownloadSource,
    register_model_group,
)

# Using constants in data processing
labels = [IGNORE_INDEX if mask == 0 else token_id for token_id, mask in zip(input_ids, loss_mask)]

# Checking if a model supports multimodal
from llamafactory.extras.constants import MULTIMODAL_SUPPORTED_MODELS
is_vlm = model_name in MULTIMODAL_SUPPORTED_MODELS

# Registering a new model group
register_model_group(
    models={
        "MyModel-7B": {DownloadSource.DEFAULT: "org/my-model-7b"},
        "MyModel-7B-Chat": {DownloadSource.DEFAULT: "org/my-model-7b-chat"},
    },
    template="my_template",
)

# Accessing the model registry
for name, sources in SUPPORTED_MODELS.items():
    hf_path = sources.get(DownloadSource.DEFAULT)
    template = DEFAULT_TEMPLATE.get(name, "")

Related Pages

• Hiyouga_LLaMA_Factory_Chat_Template - Templates referenced by DEFAULT_TEMPLATE mapping
• Hiyouga_LLaMA_Factory_Multimodal_Plugin - Uses IMAGE_PLACEHOLDER, VIDEO_PLACEHOLDER, AUDIO_PLACEHOLDER
• Hiyouga_LLaMA_Factory_Data_Collator - Uses IGNORE_INDEX for label masking
• Hiyouga_LLaMA_Factory_Data_Loader - Uses FILEEXT2TYPE for local file loading
• Hiyouga_LLaMA_Factory_Misc_Utils - Companion utility module in the extras package