Implementation:EvolvingLMMs Lab Lmms eval ChatMessages

Knowledge Sources	lmms-eval
Domains	Data_Processing, Multimodal
Last Updated	2026-02-14 00:00 GMT

Overview

Concrete tool for structured multimodal message handling provided by the lmms-eval framework.

Description

The ChatMessages class in lmms_eval/protocol.py is a Pydantic model that encapsulates a list of ChatMessage objects. Each message has a role and a list of typed content items. The class provides methods to extract media objects, convert messages to HuggingFace format, convert to OpenAI API format with base64-encoded images, and encode individual images.

Tasks that follow the chat protocol return ChatMessages from their doc_to_messages function. The model wrapper then calls the appropriate conversion method based on its backend.

Usage

Use ChatMessages when implementing a chat-protocol model wrapper. Call extract_media() to get separate media lists if the model handles text and media separately. Call to_hf_messages() for models using HuggingFace processor APIs. Call to_openai_messages() for OpenAI-compatible APIs.

Code Reference

Source Location

• Repository: lmms-eval
• File: lmms_eval/protocol.py
• Lines: L45-178

Signature

class ChatMessages(BaseModel):
    messages: List[ChatMessage]

    def extract_media(self) -> tuple[list, list, list]: ...

    def to_hf_messages(self, video_kwargs: Dict[str, str] = None) -> list[dict]: ...

    def to_openai_messages(self, video_kwargs: Dict[str, str] = {}) -> list[dict]: ...

    def encode_image(self, image: Union[Image.Image, str]) -> str: ...

Content Type Definitions

class ChatTextContent(BaseModel):
    type: Literal["text"] = "text"
    text: str

class ChatImageContent(BaseModel):
    type: Literal["image"] = "image"
    url: Any

class ChatVideoContent(BaseModel):
    type: Literal["video"] = "video"
    url: Any

class ChatAudioContent(BaseModel):
    type: Literal["audio"] = "audio"
    url: Any

ChatContent = Union[ChatTextContent, ChatImageContent, ChatVideoContent, ChatAudioContent]

class ChatMessage(BaseModel):
    role: Literal["user", "system", "assistant"]
    content: List[ChatContent]

Import

from lmms_eval.protocol import (
    ChatMessages,
    ChatMessage,
    ChatTextContent,
    ChatImageContent,
    ChatVideoContent,
    ChatAudioContent,
)

I/O Contract

Inputs

Name	Type	Required	Description
messages	List[ChatMessage]	Yes	List of chat messages, each containing a role and list of typed content items (text, image, video, audio).
video_kwargs	Dict[str, str]	No	Video processing parameters passed to conversion methods. Common key: "nframes" (default 32) controlling frame extraction count.
image	Union[PIL.Image.Image, str]	Yes (for encode_image)	A PIL Image object or a file path string to encode as base64 PNG.

Outputs

Name	Type	Description
extract_media()	tuple[list, list, list]	Three lists: images, videos, audios. Each list contains the raw url values from the corresponding content types across all messages.
to_hf_messages()	list[dict]	HuggingFace processor-compatible message dicts. Each dict has "role" and "content" keys. Content items use type-specific keys: {"type": "image", "image": url}, {"type": "video", "video": url, ...video_kwargs}, {"type": "audio", "audio": url}, {"type": "text", "text": str}.
to_openai_messages()	list[dict]	OpenAI API-compatible message dicts. Images are base64-encoded under "image_url". Videos are frame-extracted and each frame is base64-encoded as a separate image entry. Audio uses "audio_url".
encode_image()	str	Base64-encoded PNG string of the input image.

Usage Examples

Extracting Media from Messages

from lmms_eval.protocol import (
    ChatMessages, ChatMessage,
    ChatTextContent, ChatImageContent, ChatVideoContent,
)
from PIL import Image

messages = ChatMessages(messages=[
    ChatMessage(role="user", content=[
        ChatImageContent(url=Image.open("photo.jpg")),
        ChatTextContent(text="Describe this image."),
    ])
])

images, videos, audios = messages.extract_media()
# images = [<PIL.Image.Image>]
# videos = []
# audios = []

Converting to HuggingFace Format

hf_msgs = messages.to_hf_messages(video_kwargs={"nframes": 16})
# [
#     {
#         "role": "user",
#         "content": [
#             {"type": "image", "image": <PIL.Image.Image>},
#             {"type": "text", "text": "Describe this image."},
#         ]
#     }
# ]

Converting to OpenAI Format

openai_msgs = messages.to_openai_messages()
# [
#     {
#         "role": "user",
#         "content": [
#             {"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBOR..."}},
#             {"type": "text", "text": "Describe this image."},
#         ]
#     }
# ]

Using in a Chat Model Wrapper

class MyChatModel(lmms):
    is_simple = False

    def generate_until(self, requests):
        results = []
        for req in requests:
            ctx, doc_to_messages, gen_kwargs, doc_id, task, split = req.args
            chat_messages = doc_to_messages(req.doc)

            # Choose conversion based on backend
            hf_messages = chat_messages.to_hf_messages()
            inputs = self.processor.apply_chat_template(
                hf_messages, add_generation_prompt=True, return_tensors="pt"
            )
            output = self.model.generate(**inputs)
            results.append(self.processor.decode(output[0]))
        return results

Related Pages

Implements Principle

• Principle:EvolvingLMMs_Lab_Lmms_eval_Media_Handling