Implementation:NVIDIA NeMo Curator PromptFormatter

Knowledge Sources	NVIDIA NeMo Curator
Domains	Machine Learning, NLP, Vision Language Models
Last Updated	2026-02-14 00:00 GMT

Overview

Formats text prompts with video inputs into the structured input format expected by vision-language models, specifically Qwen2.5-VL.

Description

The PromptFormatter class bridges the gap between raw prompt strings and video data and the specific input format required by VLM inference engines (such as vLLM with Qwen). It uses a variant mapping system (currently supporting only "qwen" which maps to Qwen/Qwen2.5-VL-7B-Instruct) to load the appropriate HuggingFace AutoProcessor.

The generate_inputs method creates a chat-style message structure containing video and text content, applies the processor's chat template to produce a formatted prompt string, and returns a dictionary with "prompt" and "multi_modal_data" keys. The text prompt is cached after the first generation for efficiency, unless override_text_prompt=True is specified to force regeneration.

The create_message method constructs the message list in the format expected by the processor's chat template, with a single user message containing a video element and a text element.

Usage

Use PromptFormatter in the video captioning pipeline when you need to prepare inputs for a vision-language model. It is used by QwenVL and related components to format prompts before sending them to vLLM for inference.

Code Reference

Source Location

Repository: NeMo-Curator
File: nemo_curator/models/prompt_formatter.py
Lines: 1-99

Signature

VARIANT_MAPPING = {
    "qwen": "Qwen/Qwen2.5-VL-7B-Instruct",
}

class PromptFormatter:
    def __init__(self, prompt_variant: str): ...
    def generate_inputs(
        self,
        prompt: str,
        video_inputs: torch.Tensor | None = None,
        *,
        override_text_prompt: bool = False,
    ) -> dict[str, Any]: ...
    def create_message(self, prompt: str) -> list[dict[str, Any]]: ...

Import

from nemo_curator.models.prompt_formatter import PromptFormatter

I/O Contract

Inputs (Constructor)

Name	Type	Required	Description
prompt_variant	str	Yes	Model variant identifier (currently only "qwen" is supported). Raises ValueError for invalid variants.

Inputs (generate_inputs)

Name	Type	Required	Description
prompt	str	Yes	Text prompt to include with the input
video_inputs	torch.Tensor or None	No	Pre-processed video tensor. Passed through as multi_modal_data.
override_text_prompt	bool	No	If True, regenerates the formatted text prompt instead of using the cached version (default: False)

Outputs (generate_inputs)

Name	Type	Description
result	dict[str, Any]	Dictionary containing "prompt" (formatted text string with chat template applied) and "multi_modal_data" (dict with "video" key containing the video tensor)

Variant Mapping

Variant Key	HuggingFace Model ID
"qwen"	Qwen/Qwen2.5-VL-7B-Instruct

Message Format

The create_message method generates the following structure:

[
    {
        "role": "user",
        "content": [
            {"type": "video"},
            {"type": "text", "text": prompt},
        ],
    },
]

Usage Examples

Basic Usage

import torch
from nemo_curator.models.prompt_formatter import PromptFormatter

formatter = PromptFormatter(prompt_variant="qwen")

# Generate inputs for a video captioning task
video_tensor = torch.randn(16, 3, 224, 224)  # 16 frames
inputs = formatter.generate_inputs(
    prompt="Describe the content of this video in detail.",
    video_inputs=video_tensor,
)

print(inputs["prompt"])       # Formatted chat template string
print(inputs["multi_modal_data"]["video"].shape)  # Video tensor

Override Cached Prompt

from nemo_curator.models.prompt_formatter import PromptFormatter

formatter = PromptFormatter(prompt_variant="qwen")

# First call caches the text prompt
inputs1 = formatter.generate_inputs(prompt="Describe this video.")

# Second call reuses cached prompt unless override is set
inputs2 = formatter.generate_inputs(
    prompt="What objects appear in this video?",
    override_text_prompt=True,
)

Related Pages

Environment:NVIDIA_NeMo_Curator_Python_Linux_Base
NVIDIA_NeMo_Curator_QwenVL -- Vision-language model that uses PromptFormatter

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment