Implementation:Vllm project Vllm ImageAsset And VideoAsset

Knowledge Sources	vLLM vLLM Docs
Domains	Image Processing, Video Processing, Multimodal AI
Last Updated	2026-02-08 13:00 GMT

Overview

Concrete tool for loading images and video frames into the required data formats for VLM inference, provided by vLLM's asset utilities and standard Python libraries.

Description

vLLM provides two dataclass-based asset loaders for test and example images/videos, along with a standalone video_to_ndarrays function for arbitrary video files:

• ImageAsset: A frozen dataclass that loads pre-registered test images from vLLM's public S3-hosted asset store. It exposes a .pil_image property returning a PIL.Image.Image object. Available image names include "stop_sign", "cherry_blossom", "hato", and several others.
• VideoAsset: A frozen dataclass that downloads and caches test videos from HuggingFace, then provides frame extraction via .np_ndarrays (returning a NumPy array of shape (num_frames, H, W, 3)) and .pil_images (returning a list of PIL images). It also provides .metadata for video metadata needed by certain models.
• video_to_ndarrays: A standalone function that extracts uniformly sampled frames from any video file path using OpenCV, returning an RGB NumPy array.

For production use with custom images and videos, users should use PIL.Image.open(path) for images and video_to_ndarrays(path, num_frames) for videos.

Usage

Use these tools when:

• Loading test images for prototyping VLM pipelines with vLLM.
• Extracting video frames for video understanding tasks.
• Preparing custom images or videos for offline VLM inference.

Code Reference

Source Location

• Repository: vllm
• File: vllm/assets/image.py (lines 32-62), vllm/assets/video.py (lines 44-73, 111-149)

Signature

# ImageAsset
@dataclass(frozen=True)
class ImageAsset:
    name: ImageAssetName  # Literal["stop_sign", "cherry_blossom", "hato", ...]

    @property
    def pil_image(self) -> PIL.Image.Image: ...
    def pil_image_ext(self, ext: str) -> PIL.Image.Image: ...
    @property
    def image_embeds(self) -> torch.Tensor: ...

# VideoAsset
@dataclass(frozen=True)
class VideoAsset:
    name: VideoAssetName  # Literal["baby_reading"]
    num_frames: int = -1  # -1 means all frames

    @property
    def np_ndarrays(self) -> npt.NDArray: ...
    @property
    def pil_images(self) -> list[PIL.Image.Image]: ...
    @property
    def metadata(self) -> dict[str, Any]: ...

# Standalone video utility
def video_to_ndarrays(path: str, num_frames: int = -1) -> npt.NDArray: ...

Import

from vllm.assets.image import ImageAsset
from vllm.assets.video import VideoAsset, video_to_ndarrays

I/O Contract

Inputs

Name	Type	Required	Description
name (ImageAsset)	ImageAssetName	Yes	Name of the pre-registered test image (e.g., "cherry_blossom", "stop_sign")
name (VideoAsset)	VideoAssetName	Yes	Name of the pre-registered test video (e.g., "baby_reading")
num_frames	int	No	Number of frames to extract from video; -1 extracts all frames (default: -1)
path (video_to_ndarrays)	str	Yes	File path to any video file for custom video loading

Outputs

Name	Type	Description
pil_image	PIL.Image.Image	PIL Image object ready for VLM input (from ImageAsset)
np_ndarrays	npt.NDArray	NumPy array of shape (num_frames, H, W, 3) in RGB order (from VideoAsset or video_to_ndarrays)
pil_images	list[PIL.Image.Image]	List of PIL Image objects, one per video frame
metadata	dict[str, Any]	Video metadata including total_num_frames, fps, duration, video_backend

Usage Examples

Loading a Test Image

from vllm.assets.image import ImageAsset
from vllm.multimodal.image import convert_image_mode

# Load a pre-registered test image
image = ImageAsset("cherry_blossom").pil_image
image = convert_image_mode(image, "RGB")

print(type(image))  # <class 'PIL.Image.Image'>
print(image.size)   # (width, height)

Loading a Custom Image from Disk

from PIL import Image

# Load any image from a file path
image = Image.open("/path/to/my_image.jpg").convert("RGB")

Extracting Video Frames from a Test Video

from vllm.assets.video import VideoAsset

# Extract 16 uniformly sampled frames from the test video
video_asset = VideoAsset(name="baby_reading", num_frames=16)
frames = video_asset.np_ndarrays   # shape: (16, H, W, 3)
metadata = video_asset.metadata    # dict with fps, duration, etc.

print(frames.shape)  # (16, height, width, 3)

Extracting Frames from a Custom Video

from vllm.assets.video import video_to_ndarrays

# Extract 8 frames from any video file
frames = video_to_ndarrays("/path/to/my_video.mp4", num_frames=8)
print(frames.shape)  # (8, height, width, 3) in RGB order

Related Pages

Implements Principle

• Principle:Vllm_project_Vllm_Multimodal_Input_Preparation