Environment:OpenGVLab InternVL PEFT LoRA
| Knowledge Sources | |
|---|---|
| Domains | Infrastructure, Parameter_Efficient_Finetuning |
| Last Updated | 2026-02-07 14:00 GMT |
Overview
PEFT (Parameter-Efficient Fine-Tuning) >= 0.4.0 library providing LoRA adapter injection for both the vision encoder and language model components of InternVL.
Description
The PEFT library is used to inject Low-Rank Adaptation (LoRA) adapters into InternVL models for parameter-efficient fine-tuning. InternVL supports LoRA on two components independently: the vision encoder (InternViT backbone) and the language model (InternLM2/LLaMA/Phi-3/Qwen2). The LoRA configuration uses a fixed convention of `lora_alpha = 2 * rank` and targets specific attention projection layers. After training, LoRA adapters can be merged back into the base model weights.
Usage
Use this environment for LoRA fine-tuning workflows where full model fine-tuning is too memory-intensive. Enabled by setting `--use_backbone_lora` and/or `--use_llm_lora` command-line arguments to a non-zero rank value. Also required for the LoRA merging tool.
System Requirements
| Category | Requirement | Notes |
|---|---|---|
| Hardware | Same as PyTorch_CUDA environment | LoRA reduces VRAM requirements vs full fine-tuning |
| VRAM | Reduced compared to full fine-tuning | Exact savings depend on rank and target modules |
Dependencies
Python Packages
- `peft` >= 0.4.0
- `torch` >= 2.0 (prerequisite)
- `transformers` == 4.37.2 (prerequisite)
Credentials
No credentials required.
Quick Install
pip install peft>=0.4.0
Code Evidence
PEFT import from `modeling_internvl_chat.py:16`:
from peft import LoraConfig, get_peft_model
Vision encoder LoRA injection from `modeling_internvl_chat.py:111-119`:
def wrap_backbone_lora(self, r=128, lora_alpha=256, lora_dropout=0.05):
lora_config = LoraConfig(
r=r,
target_modules=['attn.qkv', 'attn.proj', 'mlp.fc1', 'mlp.fc2'],
lora_alpha=lora_alpha,
lora_dropout=lora_dropout,
)
self.vision_model = get_peft_model(self.vision_model, lora_config)
self.vision_model.print_trainable_parameters()
LLM LoRA injection from `modeling_internvl_chat.py:121-141`:
def wrap_llm_lora(self, r=128, lora_alpha=256, lora_dropout=0.05):
# Determines target modules based on LLM architecture
lora_config = LoraConfig(
r=r,
target_modules=target_modules,
lora_alpha=lora_alpha,
lora_dropout=lora_dropout,
task_type='CAUSAL_LM',
)
self.language_model = get_peft_model(self.language_model, lora_config)
self.language_model.enable_input_require_grads()
self.language_model.print_trainable_parameters()
Alpha = 2 * rank convention from `internvl_chat_finetune.py:1003-1008`:
if model_args.use_backbone_lora:
model.wrap_backbone_lora(r=model_args.use_backbone_lora,
lora_alpha=2 * model_args.use_backbone_lora)
if model_args.use_llm_lora:
model.wrap_llm_lora(r=model_args.use_llm_lora,
lora_alpha=2 * model_args.use_llm_lora)
Common Errors
| Error Message | Cause | Solution |
|---|---|---|
| `ModuleNotFoundError: No module named 'peft'` | PEFT not installed | `pip install peft>=0.4.0` |
| `ValueError: target_modules not found in model` | Wrong target module names for LLM architecture | Check that target_modules match the LLM type (InternLM2 vs LLaMA vs Phi-3 vs Qwen2) |
| LoRA weights not saving correctly | DeepSpeed ZeRO Stage 3 conflict with PEFT | Use `save_pretrained()` with DeepSpeed gather |
Compatibility Notes
- Architecture-specific targets: Different LLM backends have different LoRA target modules. InternLM2 targets `wqkv`, `wo`, `w1`, `w2`, `w3`. LLaMA targets `q_proj`, `k_proj`, `v_proj`, etc. The `wrap_llm_lora` method auto-selects based on `llm_arch_name`.
- Independent LoRA: Vision encoder and language model LoRA are configured independently. You can use LoRA on one, both, or neither.
- Merge tool: The `tools/merge_lora.py` script merges adapter weights back into the base model for deployment.