Implementation:OpenGVLab InternVL LLaVA Conversation

Knowledge Sources	OpenGVLab_InternVL
Domains	Prompt Engineering, Conversation Management, Multi-turn Dialogue
Last Updated	2026-02-07 14:00 GMT

Overview

Conversation template management system that formats multi-turn dialogues for different LLM backends, supporting over 15 predefined templates including Vicuna, LLaMA-2, MPT, InternLM, InternVL-ZH, Hermes-2, and Phi-3.

Description

This module defines the conversation template infrastructure used across the LLaVA pipeline to ensure correct prompt formatting for each supported language model architecture.

The SeparatorStyle enum defines seven distinct formatting styles: SINGLE (single separator like "###"), TWO (alternating separators like " " and ""), MPT (im_start/im_end format), PLAIN (no role prefix), LLAMA_2 (instruction wrapping with <<SYS>> and [INST] tags), CHATINTERN (InternLM's /<eoh>/<eoa> format), and INTERNVL_ZH (InternVL's human/bot Chinese format).

The Conversation dataclass maintains conversation state including system prompt, role names, message history, separator style, and stop criteria. Its get_prompt() method generates the full formatted prompt string by iterating over messages and applying the appropriate separator style. The method also handles multimodal image tokens, inserting <image> placeholders and supporting the mmtag variant format.

Additional methods include get_images() for extracting and processing images from conversation history (with support for pad, resize, crop modes), to_gradio_chatbot() for converting to Gradio-compatible format, copy() for creating conversation snapshots, and dict() for serialization.

The module provides 15+ pre-defined conversation templates registered in the conv_templates dictionary, accessible by name (e.g., "vicuna_v1", "llama_2", "mpt", "internlm2-chat", "phi3-chat").

Usage

Use this module to create and manage conversation contexts when building prompts for different LLM backends. Look up a template by name from conv_templates, copy it, append messages, then call get_prompt() to get the formatted string for tokenization.

Code Reference

Source Location

• Repository: OpenGVLab_InternVL
• File: internvl_chat_llava/llava/conversation.py
• Lines: 1-479

Signature

class SeparatorStyle(Enum):
    SINGLE = auto()
    TWO = auto()
    MPT = auto()
    PLAIN = auto()
    LLAMA_2 = auto()
    CHATINTERN = auto()
    INTERNVL_ZH = auto()

@dataclasses.dataclass
class Conversation:
    system: str
    roles: List[str]
    messages: List[List[str]]
    offset: int
    sep_style: SeparatorStyle = SeparatorStyle.SINGLE
    sep: str = "###"
    sep2: str = None
    version: str = "Unknown"
    stop_str: Union[str, List[str]] = None
    stop_token_ids: List[int] = None

    def get_prompt(self) -> str: ...
    def append_message(self, role, message): ...
    def get_images(self, return_pil=False, return_org=False) -> list: ...
    def to_gradio_chatbot(self) -> list: ...
    def copy(self) -> 'Conversation': ...
    def dict(self) -> dict: ...

conv_templates: dict  # Maps template names to Conversation instances

Import

from llava.conversation import conv_templates, SeparatorStyle

I/O Contract

Inputs

Name	Type	Required	Description
system	str	Yes	System prompt message
roles	List[str]	Yes	Role names, e.g., ("USER", "ASSISTANT")
messages	List[List[str]]	Yes	List of [role, message] pairs
sep_style	SeparatorStyle	Yes	Enum selecting the formatting style
sep	str	Yes	Primary separator string
sep2	str	No	Secondary separator for TWO/CHATINTERN styles

Outputs

Name	Type	Description
prompt	str	Fully formatted conversation prompt ready for tokenization (from get_prompt)
images	list	Extracted images as PIL Images or base64 strings (from get_images)
chatbot	list	Gradio-compatible [[user, assistant], ...] pairs (from to_gradio_chatbot)

Usage Examples

Basic Usage

from llava.conversation import conv_templates

# Get a conversation template for Vicuna v1
conv = conv_templates["vicuna_v1"].copy()
conv.append_message(conv.roles[0], "What is in this image?")
conv.append_message(conv.roles[1], None)

# Get the formatted prompt
prompt = conv.get_prompt()
# Output: "A chat between a curious user... USER: What is in this image? ASSISTANT:"

Related Pages

• Principle:OpenGVLab_InternVL_Conversation_Template_System