Implementation:Vllm project Vllm LLM Generate Multimodal

Knowledge Sources	vLLM vLLM Docs
Domains	Text Generation, Vision Language Models, Batch Inference
Last Updated	2026-02-08 13:00 GMT

Overview

Concrete tool for running multimodal text generation with visual inputs through vLLM's LLM.generate() method, provided by vLLM.

Description

The LLM.generate() method is the primary interface for offline multimodal inference. For VLM tasks, prompts are passed as dictionaries containing both the formatted text prompt and the visual data. The method automatically batches requests, manages GPU memory through continuous batching, preprocesses multimodal inputs through the model's processor pipeline, and returns structured RequestOutput objects.

The method accepts either a single prompt dictionary or a list of prompt dictionaries for batch inference. Each prompt dictionary must contain:

• "prompt": The formatted text string with vision token placeholders.
• "multi_modal_data": A dictionary mapping modality names ("image", "video") to data objects (PIL.Image.Image, np.ndarray, or lists thereof).

Optionally, "multi_modal_uuids" can be included to enable multimodal preprocessor caching across requests that share the same visual input.

Usage

Use LLM.generate() for multimodal tasks when:

• Running offline VLM inference on one or more image/video inputs.
• Batch processing visual question answering, captioning, or OCR tasks.
• Integrating VLM inference into data processing pipelines.

Code Reference

Source Location

• Repository: vllm
• File: vllm/entrypoints/llm.py (lines 396-459)

Signature

class LLM:
    def generate(
        self,
        prompts: PromptType | Sequence[PromptType],
        sampling_params: SamplingParams | Sequence[SamplingParams] | None = None,
        *,
        use_tqdm: bool | Callable[..., tqdm] = True,
        lora_request: list[LoRARequest] | LoRARequest | None = None,
        priority: list[int] | None = None,
        tokenization_kwargs: dict[str, Any] | None = None,
    ) -> list[RequestOutput]: ...

Import

from vllm import LLM, SamplingParams

I/O Contract

Inputs

Name	Type	Required	Description
prompts	list[dict]	Yes	Prompt dictionary(ies) with "prompt" (str) and "multi_modal_data" (dict) keys
prompts["prompt"]	str	Yes	Formatted text prompt with vision token placeholders
prompts["multi_modal_data"]	dict[str, Any]	Yes	Mapping of modality name to data: {"image": PIL.Image} or {"video": np.ndarray}
prompts["multi_modal_uuids"]	dict[str, str]	No	UUID for multimodal cache hit/miss tracking
sampling_params	None	No	Sampling configuration; if None, uses model defaults
lora_request	None	No	LoRA adapter request (e.g., for Phi-4-multimodal vision LoRA)

Outputs

Name	Type	Description
outputs	list[RequestOutput]	List of RequestOutput objects, one per input prompt, in the same order as inputs

Usage Examples

Single Image Inference

from vllm import LLM, SamplingParams
from vllm.assets.image import ImageAsset

# Initialize the model
llm = LLM(
    model="llava-hf/llava-1.5-7b-hf",
    max_model_len=4096,
    limit_mm_per_prompt={"image": 1},
)

# Prepare image and prompt
image = ImageAsset("cherry_blossom").pil_image
prompt = "USER: <image>\nWhat is the content of this image?\nASSISTANT:"

# Run inference
outputs = llm.generate(
    {"prompt": prompt, "multi_modal_data": {"image": image}},
    sampling_params=SamplingParams(temperature=0, max_tokens=128),
)

print(outputs[0].outputs[0].text)

Batch Image Inference

from vllm import LLM, SamplingParams
from PIL import Image

llm = LLM(
    model="llava-hf/llava-1.5-7b-hf",
    max_model_len=4096,
    limit_mm_per_prompt={"image": 1},
)

image = Image.open("/path/to/image.jpg").convert("RGB")

# Create batch of prompts with the same image
prompts = [
    {
        "prompt": "USER: <image>\nWhat is in this image?\nASSISTANT:",
        "multi_modal_data": {"image": image},
    },
    {
        "prompt": "USER: <image>\nDescribe this image in detail.\nASSISTANT:",
        "multi_modal_data": {"image": image},
    },
]

sampling_params = SamplingParams(temperature=0.2, max_tokens=64)
outputs = llm.generate(prompts, sampling_params=sampling_params)

for output in outputs:
    print(output.outputs[0].text)

Video Inference

from vllm import LLM, SamplingParams
from vllm.assets.video import VideoAsset

llm = LLM(
    model="llava-hf/LLaVA-NeXT-Video-7B-hf",
    max_model_len=8192,
    max_num_seqs=2,
    limit_mm_per_prompt={"video": 1},
)

video_frames = VideoAsset(name="baby_reading", num_frames=16).np_ndarrays

outputs = llm.generate(
    {
        "prompt": "USER: <video>\nWhy is this video funny? ASSISTANT:",
        "multi_modal_data": {"video": video_frames},
    },
    sampling_params=SamplingParams(temperature=0, max_tokens=128),
)

print(outputs[0].outputs[0].text)

Inference with Multimodal UUID Caching

from vllm import LLM, SamplingParams
from vllm.assets.image import ImageAsset

llm = LLM(
    model="llava-hf/llava-1.5-7b-hf",
    max_model_len=4096,
    limit_mm_per_prompt={"image": 1},
)

image = ImageAsset("cherry_blossom").pil_image

# First request with UUID
outputs_1 = llm.generate(
    {
        "prompt": "USER: <image>\nWhat is this?\nASSISTANT:",
        "multi_modal_data": {"image": image},
        "multi_modal_uuids": {"image": "uuid_cherry"},
    },
    sampling_params=SamplingParams(temperature=0, max_tokens=64),
)

# Second request reusing cached image via UUID (pass None for data)
outputs_2 = llm.generate(
    {
        "prompt": "USER: <image>\nDescribe the colors.\nASSISTANT:",
        "multi_modal_data": {"image": None},
        "multi_modal_uuids": {"image": "uuid_cherry"},
    },
    sampling_params=SamplingParams(temperature=0, max_tokens=64),
)

Related Pages

Implements Principle

• Principle:Vllm_project_Vllm_Multimodal_Generation