Implementation:Run llama Llama index MultiModalLLM

Overview

This module defines the base interface for multi-modal language models in LlamaIndex. It includes the MultiModalLLMMetadata configuration class and the abstract MultiModalLLM base class, which provides a unified API for LLMs that can process both text and image inputs.

Source file: llama-index-core/llama_index/core/multi_modal_llms/base.py (182 lines)

Class Hierarchy

BaseModel
  └── MultiModalLLMMetadata

BaseComponent, DispatcherSpanMixin
  └── MultiModalLLM

MultiModalLLMMetadata

class MultiModalLLMMetadata(BaseModel):

A Pydantic model describing the capabilities and configuration of a multi-modal LLM.

Field	Type	Default	Description
context_window	Optional[int]	DEFAULT_CONTEXT_WINDOW	Total tokens the model can accept as input when generating a response
num_output	Optional[int]	DEFAULT_NUM_OUTPUTS	Number of tokens the model can output
num_input_files	Optional[int]	DEFAULT_NUM_INPUT_FILES	Number of input files the model can process
is_function_calling_model	Optional[bool]	False	Whether the model supports function calling (e.g., OpenAI-style)
model_name	str	"unknown"	Model name for logging, testing, and identification
is_chat_model	bool	False	Whether the model exposes a chat interface (sequence of messages)

The class uses ConfigDict(protected_namespaces=("pydantic_model_",)) to avoid conflicts with the model_ prefix used by Pydantic internally.

MultiModalLLM

class MultiModalLLM(BaseComponent, DispatcherSpanMixin):

Abstract base class for multi-modal LLMs. Inherits from:

• BaseComponent -- provides component serialization infrastructure
• DispatcherSpanMixin -- provides instrumentation and tracing

Configuration

Field	Type	Description
callback_manager	CallbackManager	Callback manager for event tracking (excluded from serialization)

Uses ConfigDict(arbitrary_types_allowed=True) to allow non-Pydantic types in fields.

Abstract Property

metadata

@property
@abstractmethod
def metadata(self) -> MultiModalLLMMetadata:

Must be implemented by subclasses to return the model's metadata.

Abstract Synchronous Methods

Method	Signature	Description
complete	(prompt: str, image_documents: List[Union[ImageNode, ImageBlock]], **kwargs) -> CompletionResponse	Complete a prompt with image inputs
stream_complete	(prompt: str, image_documents: List[Union[ImageNode, ImageBlock]], **kwargs) -> CompletionResponseGen	Streaming completion with image inputs
chat	(messages: Sequence[ChatMessage], **kwargs) -> ChatResponse	Chat with message sequence
stream_chat	(messages: Sequence[ChatMessage], **kwargs) -> ChatResponseGen	Streaming chat with message sequence

Abstract Async Methods

Method	Signature	Description
acomplete	async (prompt: str, image_documents: List[Union[ImageNode, ImageBlock]], **kwargs) -> CompletionResponse	Async completion with images
astream_complete	async (prompt: str, image_documents: List[Union[ImageNode, ImageBlock]], **kwargs) -> CompletionResponseAsyncGen	Async streaming completion with images
achat	async (messages: Sequence[ChatMessage], **kwargs) -> ChatResponse	Async chat
astream_chat	async (messages: Sequence[ChatMessage], **kwargs) -> ChatResponseAsyncGen	Async streaming chat

Image Input Types

The completion methods accept image documents as List[Union[ImageNode, ImageBlock]], supporting both:

• ImageNode -- a schema node representing an image with metadata
• ImageBlock -- an image block type from the LLM types system

Automatic Callback Decoration

def __init_subclass__(cls, **kwargs: Any) -> None:

The __init_subclass__ hook automatically decorates the following methods on all subclasses:

Methods	Decorator Applied
chat, achat, stream_chat, astream_chat	@llm_chat_callback()
complete, acomplete, stream_complete, astream_complete	@llm_completion_callback()

The comment in the source explains that callback decorators must be applied before span decorators so that spans properly contain the events. The method checks if an attribute exists in the subclass's __dict__ (not inherited) and is callable before applying the decorator.

Dependencies

• llama_index.core.base.llms.types -- provides all response types (ChatMessage, ChatResponse, CompletionResponse, etc.) and ImageBlock
• llama_index.core.bridge.pydantic -- provides BaseModel, ConfigDict, Field
• llama_index.core.callbacks.CallbackManager -- callback management
• llama_index.core.constants -- provides DEFAULT_CONTEXT_WINDOW, DEFAULT_NUM_INPUT_FILES, DEFAULT_NUM_OUTPUTS
• llama_index.core.instrumentation.DispatcherSpanMixin -- instrumentation support
• llama_index.core.llms.callbacks -- provides llm_chat_callback and llm_completion_callback
• llama_index.core.schema -- provides BaseComponent and ImageNode