Implementation:Hiyouga LLaMA Factory RoPE Config

Knowledge Sources	Hiyouga_LLaMA_Factory
Domains	Positional Encoding, Context Extension
Last Updated	2026-02-06 19:00 GMT

Overview

Configures Rotary Position Embedding (RoPE) scaling strategies to extend model context length beyond original pretraining limits.

Description

The rope module provides the configure_rope function which supports four RoPE scaling strategies: linear, dynamic NTK, YaRN, and LLaMA3-style scaling. It computes the scaling factor from the ratio of the desired maximum sequence length to the original maximum position embeddings stored in the model config. For dynamic and YaRN strategies, the original max position embeddings are preserved as an additional parameter. For LLaMA3-style scaling, low-frequency and high-frequency factors (1.0 and 4.0 respectively) are additionally configured. The function updates the model configuration's max_position_embeddings and rope_scaling attributes in-place.

Usage

Use this module when you need to train or perform inference with a model at a context length exceeding its original pretraining limit. It is called automatically by patch_config in the model patcher pipeline when rope_scaling is set in the model arguments.

Code Reference

Source Location

• Repository: Hiyouga_LLaMA_Factory
• File: src/llamafactory/model/model_utils/rope.py
• Lines: 1-84

Signature

def configure_rope(
    config: "PretrainedConfig",
    model_args: "ModelArguments",
) -> None

Import

from llamafactory.model.model_utils.rope import configure_rope

I/O Contract

Inputs

Name	Type	Required	Description
config	PretrainedConfig	Yes	The model configuration object; must have rope_scaling and max_position_embeddings attributes for RoPE to be applied
model_args	ModelArguments	Yes	Contains rope_scaling (scaling strategy enum), model_max_length (target context length), and related settings

Outputs

Name	Type	Description
None	None	Modifies config in-place, setting rope_scaling dict with rope_type, factor, and strategy-specific parameters, and updating max_position_embeddings

Usage Examples

# Extending context length with dynamic NTK scaling
from llamafactory.model.model_utils.rope import configure_rope

# model_args.rope_scaling = "dynamic"
# model_args.model_max_length = 8192
# config.max_position_embeddings = 4096
configure_rope(config, model_args)
# config.max_position_embeddings is now 8192
# config.rope_scaling = {"rope_type": "dynamic", "factor": 2.0, "original_max_position_embeddings": 4096}

Related Pages

• Hiyouga_LLaMA_Factory_Model_Patcher - Calls configure_rope during patch_config