Implementation:Mit han lab Llm awq Conversation Template

Knowledge Sources	Mit_han_lab_Llm_awq
Domains	NLP, Prompt_Engineering
Last Updated	2026-02-15 00:00 GMT

Overview

Concrete tool for managing conversation prompt templates across multiple LLM chat formats provided by the tinychat InternVL module.

Description

The Conversation dataclass stores template metadata (system message, role names, separator style, stop tokens) and implements get_prompt() which formats conversation history according to one of 18 separator styles (CHATML, LLAMA2, INTERNVL_ZH, MPT, etc.). The SeparatorStyle enum defines the formatting modes. Templates are stored in a global conv_templates dictionary via register_conv_template() and retrieved with get_conv_template() which returns a deep copy. Pre-registered templates include Hermes-2, internlm2-chat, phi3-chat, and internvl2_5.

Usage

Import these classes when formatting prompts for InternVL3 inference. Use get_conv_template() to retrieve a conversation template by name, append messages, and call get_prompt() to produce the formatted prompt string expected by the model.

Code Reference

Source Location

• Repository: Mit_han_lab_Llm_awq
• File: tinychat/models/internvl/conversation.py
• Lines: 1-391

Signature

class SeparatorStyle(IntEnum):
    ADD_COLON_SINGLE = auto()
    ADD_COLON_TWO = auto()
    ADD_COLON_SPACE_SINGLE = auto()
    NO_COLON_SINGLE = auto()
    NO_COLON_TWO = auto()
    ADD_NEW_LINE_SINGLE = auto()
    LLAMA2 = auto()
    CHATGLM = auto()
    CHATML = auto()
    CHATINTERN = auto()
    DOLLY = auto()
    RWKV = auto()
    PHOENIX = auto()
    ROBIN = auto()
    FALCON_CHAT = auto()
    CHATGLM3 = auto()
    INTERNVL_ZH = auto()
    MPT = auto()

@dataclass
class Conversation:
    name: str
    system_template: str = '{system_message}'
    system_message: str = ''
    roles: Tuple[str] = ('USER', 'ASSISTANT')
    messages: List[List[str]] = ()
    offset: int = 0
    sep_style: SeparatorStyle = SeparatorStyle.ADD_COLON_SINGLE
    sep: str = '\n'
    sep2: str = None
    stop_str: Union[str, List[str]] = None
    stop_token_ids: List[int] = None

    def get_prompt(self) -> str: ...
    def set_system_message(self, system_message: str): ...
    def append_message(self, role: str, message: str): ...
    def update_last_message(self, message: str): ...
    def to_gradio_chatbot(self): ...
    def to_openai_api_messages(self): ...
    def copy(self) -> 'Conversation': ...
    def dict(self) -> Dict: ...

def register_conv_template(template: Conversation, override: bool = False) -> None: ...
def get_conv_template(name: str) -> Conversation: ...

Import

from tinychat.models.internvl.conversation import (
    Conversation, SeparatorStyle, get_conv_template, register_conv_template
)

I/O Contract

Inputs

Name	Type	Required	Description
name	str	Yes	Template name identifier (e.g., 'internvl2_5')
system_message	str	No	System prompt content
roles	Tuple[str]	No	Role names (default: ('USER', 'ASSISTANT'))
sep_style	SeparatorStyle	No	Separator formatting style
sep	str	No	Primary separator token
stop_str	Union[str, List[str]]	No	Stop criteria strings
stop_token_ids	List[int]	No	Token IDs that trigger generation stop

Outputs

Name	Type	Description
get_prompt() returns	str	Fully formatted conversation prompt
to_gradio_chatbot() returns	List[List]	Messages formatted for Gradio UI
to_openai_api_messages() returns	List[Dict]	Messages in OpenAI API format

Usage Examples

Basic Prompt Formatting

from tinychat.models.internvl.conversation import get_conv_template

# Get the InternVL 2.5 conversation template
conv = get_conv_template('internvl2_5')
conv.set_system_message('You are a helpful assistant.')

# Add messages
conv.append_message(conv.roles[0], 'Describe this image.')
conv.append_message(conv.roles[1], None)  # Placeholder for generation

# Get formatted prompt
prompt = conv.get_prompt()

Register Custom Template

from tinychat.models.internvl.conversation import (
    Conversation, SeparatorStyle, register_conv_template
)

custom_template = Conversation(
    name='custom_chat',
    system_template='<|system|>\n{system_message}',
    system_message='You are an image analysis assistant.',
    roles=('<|user|>', '<|assistant|>'),
    sep_style=SeparatorStyle.MPT,
    sep='<|end|>\n',
    stop_str='<|end|>',
)
register_conv_template(custom_template)

Related Pages

• Principle:Mit_han_lab_Llm_awq_Conversation_Template_Management