Implementation:Volcengine Verl Get Peft Model LoRA

Field	Value
Knowledge Sources	Wrapper Doc (wraps HuggingFace PEFT library)
Domains	Parameter-Efficient Fine-Tuning, LoRA, Model Adaptation
Last Updated	2026-02-07

Overview

Description

This implementation documents how verl integrates the HuggingFace PEFT (Parameter-Efficient Fine-Tuning) library to apply LoRA (Low-Rank Adaptation) to a pretrained language model during SFT training. Within the FSDPSFTTrainer._build_model_optimizer() method, when LoRA is enabled (either via config.model.lora_rank > 0 or config.model.lora_adapter_path), the trainer either loads a pre-trained LoRA adapter using PeftModel.from_pretrained() or creates a new LoRA configuration and applies it with get_peft_model(model, LoraConfig(...)).

The LoRA configuration is constructed from verl config values including lora_rank (the rank r), lora_alpha (the scaling factor), and target_modules (which transformer layers to apply LoRA to). The task_type is always set to TaskType.CAUSAL_LM and bias is set to "none". Before applying PEFT, model.enable_input_require_grads() is called to ensure gradients flow through the frozen base model to the LoRA adapters.

Usage

LoRA is configured through the verl Hydra config system:

model:
  lora_rank: 16
  lora_alpha: 32
  target_modules:
    - q_proj
    - v_proj
    - k_proj
    - o_proj
  # OR load a pre-trained adapter:
  # lora_adapter_path: /path/to/adapter

External reference: HuggingFace PEFT Documentation

Code Reference

Attribute	Detail
Source Location	verl/trainer/fsdp_sft_trainer.py, Lines 207-370 (within _build_model_optimizer)
Signature	get_peft_model(model, LoraConfig(**lora_config))
Import	from peft import LoraConfig, get_peft_model, TaskType

I/O Contract

Inputs

Parameter	Type	Description
model	PreTrainedModel	Base HuggingFace model to wrap with LoRA adapters
config.model.lora_rank	int	LoRA rank (r); set to 0 to disable LoRA
config.model.lora_alpha	int	LoRA scaling factor (alpha)
config.model.target_modules	list[str]	List of module names to apply LoRA to (e.g., ["q_proj", "v_proj"])
config.model.lora_adapter_path	str or None	Path to a pre-trained LoRA adapter to load instead of creating a new one

Outputs

Output	Type	Description
Modified model	PeftModel	The base model wrapped with LoRA adapter layers; only adapter parameters are trainable
Side effect	Gradient configuration	enable_input_require_grads() is called on the base model before wrapping

Usage Examples

Example 1: Creating a new LoRA model via PEFT

from peft import LoraConfig, get_peft_model, TaskType
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
model.enable_input_require_grads()

lora_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    bias="none",
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# Output: trainable params: 3,407,872 || all params: 497,763,328 || trainable%: 0.685

Example 2: Loading a pre-trained LoRA adapter

from peft import PeftModel, TaskType
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
model.enable_input_require_grads()

model = PeftModel.from_pretrained(
    model,
    "/path/to/lora/adapter",
    is_trainable=True,
)

# Ensure task_type is set correctly
peft_config = model.peft_config["default"]
if isinstance(peft_config.task_type, str):
    peft_config.task_type = TaskType.CAUSAL_LM

Example 3: How verl constructs the LoRA config internally

from verl.utils.py_functional import convert_to_regular_types

lora_config = {
    "task_type": TaskType.CAUSAL_LM,
    "r": self.config.model.lora_rank,           # e.g., 16
    "lora_alpha": self.config.model.lora_alpha,  # e.g., 32
    "target_modules": convert_to_regular_types(self.config.model.target_modules),
    "bias": "none",
}
self.model = get_peft_model(self.model, LoraConfig(**lora_config))
self.model = self.model.to(torch_dtype)

Related Pages

• Principle:Volcengine_Verl_LoRA_Configuration
• verl/trainer/fsdp_sft_trainer.py -- Source file containing LoRA integration
• Implementation:Volcengine_Verl_FSDPSFTTrainer_Fit -- Trainer that uses the LoRA-wrapped model
• HuggingFace PEFT Documentation -- External library reference