Implementation:OpenGVLab InternVL LlavaMptForCausalLM

Knowledge Sources	OpenGVLab_InternVL
Domains	Multimodal Models, Language Models, LLaVA
Last Updated	2026-02-07 14:00 GMT

Overview

Defines the MPT-based LLaVA model variant by combining the MPT language model backbone with multimodal vision capabilities through LLaVA mixin classes.

Description

This module provides three classes that adapt the MosaiCML MPT language model for use as a LLaVA multimodal backbone. LlavaMptConfig extends MptConfig with the model type "llava_mpt" for HuggingFace auto-registration. LlavaMptModel combines LlavaMetaModel (which provides vision tower and projector initialization) with MptModel, adding an embed_tokens wrapper that delegates to the MPT word token embedding (wte). LlavaMptForCausalLM inherits from both MptForCausalLM and LlavaMetaForCausalLM, overriding the forward method to call prepare_inputs_labels_for_multimodal() before the standard causal LM forward pass, and overriding prepare_inputs_for_generation() to propagate image tensors through the generation pipeline. The module also supports gradient checkpointing for memory-efficient training. Both the config and model classes are registered with HuggingFace's AutoConfig and AutoModelForCausalLM registries.

Usage

Use this module when you want to use MPT as the language model backbone in a LLaVA multimodal architecture, as an alternative to LLaMA-based variants.

Code Reference

Source Location

Repository: OpenGVLab_InternVL
File: internvl_chat_llava/llava/model/language_model/llava_mpt.py
Lines: 1-97

Signature

class LlavaMptConfig(MptConfig):
    model_type = "llava_mpt"

class LlavaMptModel(LlavaMetaModel, MptModel):
    config_class = LlavaMptConfig
    def __init__(self, config: MptConfig): ...
    def embed_tokens(self, x): ...

class LlavaMptForCausalLM(MptForCausalLM, LlavaMetaForCausalLM):
    config_class = LlavaMptConfig
    supports_gradient_checkpointing = True
    def forward(self, input_ids=None, past_key_values=None, attention_mask=None,
                inputs_embeds=None, labels=None, use_cache=None,
                output_attentions=None, output_hidden_states=None,
                return_dict=None, images=None): ...
    def prepare_inputs_for_generation(self, input_ids, past_key_values=None,
                                       inputs_embeds=None, **kwargs): ...

Import

from llava.model.language_model.llava_mpt import LlavaMptConfig, LlavaMptModel, LlavaMptForCausalLM

I/O Contract

Inputs

Name	Type	Required	Description
input_ids	torch.LongTensor	No	Token IDs for the input sequence
attention_mask	torch.Tensor	No	Attention mask for padding
past_key_values	Tuple[Tuple[torch.Tensor]]	No	Cached key/value tensors for autoregressive generation
inputs_embeds	torch.Tensor	No	Pre-computed input embeddings (bypasses token embedding)
labels	torch.Tensor	No	Target labels for language modeling loss
images	torch.Tensor	No	Image tensors to encode via the vision tower
use_cache	bool	No	Whether to return cached key/value states

Outputs

Name	Type	Description
output	CausalLMOutputWithPast	Standard HuggingFace causal LM output with loss, logits, and past key values

Usage Examples

Basic Usage

from transformers import AutoConfig, AutoModelForCausalLM

# Load via HuggingFace auto classes (registered as "llava_mpt")
config = AutoConfig.from_pretrained("path/to/llava_mpt_model")
model = AutoModelForCausalLM.from_pretrained("path/to/llava_mpt_model")

# Forward pass with images
output = model(input_ids=token_ids, images=image_tensor, labels=labels)

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment