Principle:EvolvingLMMs Lab Lmms eval Media Handling
| Knowledge Sources | |
|---|---|
| Domains | Data_Processing, Multimodal |
| Last Updated | 2026-02-14 00:00 GMT |
Overview
A structured multimodal message protocol provides a uniform way to handle text, images, video, and audio inputs across diverse model backends.
Description
Multimodal evaluation involves delivering heterogeneous media -- text, images, video frames, and audio -- to models that each expect different input formats. The media handling principle establishes a typed message protocol that captures all modalities in a single, model-agnostic data structure, then provides conversion methods for specific backends.
The protocol is built on a hierarchy of Pydantic models:
- ChatTextContent: Carries a text string with
type="text". - ChatImageContent: Carries an image reference (PIL Image or file path) with
type="image". - ChatVideoContent: Carries a video reference (file path or URL) with
type="video". - ChatAudioContent: Carries an audio reference with
type="audio".
These content types are combined into ChatMessage objects, each annotated with a role ("user", "system", or "assistant"). A list of ChatMessage objects forms a ChatMessages container, which is the top-level object passed from tasks to chat-protocol models.
The key design insight is separation of concerns: tasks produce a model-agnostic ChatMessages object, and the model wrapper converts it to the specific format its backend requires using one of the provided conversion methods:
- extract_media(): Walks all messages and extracts separate lists of images, videos, and audios. Useful when the model handles media objects separately from the text prompt.
- to_hf_messages(): Converts to HuggingFace's processor-compatible message format, where each content item is a dict with type-specific keys (
"image","video","audio"). - to_openai_messages(): Converts to OpenAI API format with base64-encoded images and frame-extracted video content.
- encode_image(): Helper that converts a PIL Image or file path to a base64-encoded PNG string.
Usage
Use the ChatMessages protocol when building a chat-protocol model (is_simple = False). The task's doc_to_messages function produces a ChatMessages instance, and the model wrapper converts it to the target format within generate_until or loglikelihood.
For simple-protocol models (is_simple = True), media is handled through doc_to_visual instead, and ChatMessages is not involved.
Theoretical Basis
The message protocol follows the Adapter pattern, providing a common interface that adapts to multiple target formats:
Task (doc_to_messages)
|
v
ChatMessages (model-agnostic)
|
+---> to_hf_messages() --> HuggingFace processor format
+---> to_openai_messages() --> OpenAI API format
+---> extract_media() --> Separate media lists
The content type hierarchy enforces type safety through Pydantic's literal discriminators:
ChatContent = Union[
ChatTextContent, # type="text", text: str
ChatImageContent, # type="image", url: Any (PIL Image or path)
ChatVideoContent, # type="video", url: Any (path or URL)
ChatAudioContent, # type="audio", url: Any (path or URL)
]
ChatMessage:
role: "user" | "system" | "assistant"
content: List[ChatContent]
ChatMessages:
messages: List[ChatMessage]
For video processing, the OpenAI conversion extracts individual frames using qwen_vl_utils.fetch_video, converts each frame to a PIL Image, and encodes it as a base64 PNG. This enables API-based models that only accept images to process video content frame-by-frame.