Implementation:Hiyouga LLaMA Factory Attention Config

Knowledge Sources	Hiyouga_LLaMA_Factory
Domains	Attention Mechanisms, Model Configuration
Last Updated	2026-02-06 19:00 GMT

Overview

Configures and reports the attention implementation strategy for pretrained models, mapping user preferences to HuggingFace's internal attention backend settings.

Description

This module provides two functions for managing attention implementations. configure_attn_implementation translates user-specified attention preferences (auto, disabled/eager, SDPA, FlashAttention-2, FlashAttention-3) into the appropriate HuggingFace config attributes. It includes model-specific handling for architectures with special requirements: Gemma 2 requires FlashAttention-2 for soft-capping support, GPT-OSS uses FlashAttention-3 with hub kernels, InternLM2 uses a non-standard config attribute, and Kimi-VL and Youtu-VL require per-submodel configuration. print_attn_implementation inspects the loaded model configuration to report which attention backend is actually in use.

Usage

Use configure_attn_implementation during model config patching (called automatically by the model loader) to set the desired attention backend. Use print_attn_implementation after model loading to verify which attention implementation was selected.

Code Reference

Source Location

• Repository: Hiyouga_LLaMA_Factory
• File: src/llamafactory/model/model_utils/attention.py
• Lines: 1-115

Signature

def configure_attn_implementation(
    config: "PretrainedConfig",
    model_args: "ModelArguments",
) -> None:
    ...

def print_attn_implementation(
    config: "PretrainedConfig",
) -> None:
    ...

Import

from llamafactory.model.model_utils.attention import configure_attn_implementation, print_attn_implementation

I/O Contract

Inputs

Name	Type	Required	Description
config	PretrainedConfig	Yes	Model configuration object to modify with the attention implementation setting
model_args	ModelArguments	Yes (for configure)	Model arguments containing flash_attn setting (AttentionFunction enum: AUTO, DISABLED, SDPA, FA2, FA3)

Outputs

Name	Type	Description
(side effect)	None	Modifies config._attn_implementation (or model-specific equivalent) in-place
(side effect)	None	May modify model_args.flash_attn in-place for model-specific overrides (e.g., Gemma 2)

Usage Examples

from llamafactory.model.model_utils.attention import configure_attn_implementation, print_attn_implementation

# Configure attention for FlashAttention-2
configure_attn_implementation(config, model_args)

# After model loading, verify the attention backend
print_attn_implementation(config)
# Output: "Using FlashAttention-2 for faster training and inference."
# or:    "Using torch SDPA for faster training and inference."
# or:    "Using vanilla attention implementation."

Related Pages

• Hiyouga_LLaMA_Factory_Model_Loader - Calls configure_attn_implementation during model config patching
• Hiyouga_LLaMA_Factory_Sequence_Packing - Sequence packing relies on FlashAttention's variable-length API