Implementation:OpenGVLab InternVL Wrap LLM LoRA

Knowledge Sources	InternVL PEFT
Domains	Parameter_Efficient_Finetuning, NLP
Last Updated	2026-02-07 00:00 GMT

Overview

Concrete tool for injecting LoRA adapters into the language model component of InternVL provided by the InternVL model class.

Description

The wrap_llm_lora method on InternVLChatModel wraps the language model submodule with PEFT LoRA adapters. It automatically selects target modules based on the LLM architecture (InternLM2 vs Qwen2/LLaMA) and replaces the language_model with a PeftModel.

Usage

This method is called during training initialization when use_llm_lora > 0 in ModelArguments. It is invoked after model loading but before the training loop begins.

Code Reference

Source Location

Repository: InternVL
File: internvl_chat/internvl/model/internvl_chat/modeling_internvl_chat.py
Lines: L121-141

Signature

def wrap_llm_lora(self, r=128, lora_alpha=256, lora_dropout=0.05):
    """
    Wrap the language model with LoRA adapters.

    Args:
        r: int - LoRA rank (default 128, typical fine-tuning uses 16)
        lora_alpha: int - LoRA scaling factor (convention: 2 * r)
        lora_dropout: float - Dropout on LoRA path (default 0.05)

    Target modules (auto-selected by architecture):
        InternLM2: ['attention.wqkv', 'attention.wo',
                     'feed_forward.w1', 'feed_forward.w2', 'feed_forward.w3']
        Qwen2/LLaMA: ['self_attn.q_proj', 'self_attn.k_proj',
                       'self_attn.v_proj', 'self_attn.o_proj',
                       'mlp.gate_proj', 'mlp.down_proj', 'mlp.up_proj']
    """

Import

from internvl.model.internvl_chat import InternVLChatModel
# Called as: model.wrap_llm_lora(r=16, lora_alpha=32)

I/O Contract

Inputs

Name	Type	Required	Description
r	int	No	LoRA rank (default 128; typical usage: 16)
lora_alpha	int	No	Scaling factor (default 256; convention: 2*r)
lora_dropout	float	No	Dropout probability (default 0.05)

Outputs

Name	Type	Description
self.language_model	PeftModel	Language model wrapped with LoRA adapters; only LoRA parameters are trainable

Usage Examples

Inject LoRA with Rank 16

import torch
from internvl.model.internvl_chat import InternVLChatModel

# Load base model
model = InternVLChatModel.from_pretrained(
    'OpenGVLab/InternVL2_5-8B',
    torch_dtype=torch.bfloat16,
)

# Freeze base LLM, then inject LoRA
for param in model.language_model.parameters():
    param.requires_grad = False

model.wrap_llm_lora(r=16, lora_alpha=32, lora_dropout=0.05)
model.config.use_llm_lora = 16

# Verify: only LoRA params are trainable
trainable = sum(p.numel() for p in model.language_model.parameters() if p.requires_grad)
total = sum(p.numel() for p in model.language_model.parameters())
print(f'Trainable: {trainable:,} / {total:,} ({100*trainable/total:.2f}%)')

Related Pages

Implements Principle

Principle:OpenGVLab_InternVL_LoRA_Adapter_Injection

Requires Environment

Uses Heuristic

Heuristic:OpenGVLab_InternVL_LoRA_Alpha_Scaling

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment