Implementation:Haotian liu LLaVA Eval Model RunLLaVA

Overview

CLI tool for running single-image visual question answering inference for model validation.

Type

API Doc

Description

eval_model() in run_llava.py provides a complete single-image inference pipeline: loads the model via load_pretrained_model(), loads and processes the image, constructs a conversation prompt with the appropriate template, tokenizes with image token placeholders, and generates a response via model.generate().

The function handles:

• Model loading -- Supports both merged models and unmerged LoRA models (via --model-base)
• Image loading -- Supports local file paths and HTTP/HTTPS URLs
• Conversation template -- Auto-detects the correct template based on model name (llava_v1, llava_llama_2, mistral_instruct, mpt, chatml_direct)
• Image token injection -- Automatically prepends <image> token to the query if not already present
• Multi-image support -- Multiple images can be passed via comma-separated paths

Source

• llava/eval/run_llava.py:L50-128 (eval_model function)
• llava/eval/run_llava.py:L28-39 (image loading helpers)
• llava/eval/run_llava.py:L131-145 (CLI argument parser)

Signature

def eval_model(args) -> None:
    """Run single-image visual question answering inference.

    Args:
        args.model_path (str): Model checkpoint path or HuggingFace model ID
        args.model_base (str): Base model path (required for unmerged LoRA)
        args.image_file (str): Image path or URL (comma-separated for multiple)
        args.query (str): Text question to ask about the image
        args.conv_mode (str): Conversation template (auto-detected if None)
        args.temperature (float): Sampling temperature (default: 0.2)
        args.top_p (float): Top-p sampling parameter (default: None)
        args.num_beams (int): Number of beams for beam search (default: 1)
        args.max_new_tokens (int): Maximum tokens to generate (default: 512)
        args.sep (str): Separator for multiple image paths (default: ",")
    """

Core Inference Code

def eval_model(args):
    disable_torch_init()

    model_name = get_model_name_from_path(args.model_path)
    tokenizer, model, image_processor, context_len = load_pretrained_model(
        args.model_path, args.model_base, model_name
    )

    qs = args.query
    # Auto-prepend <image> token if not present
    if IMAGE_PLACEHOLDER in qs:
        if model.config.mm_use_im_start_end:
            qs = re.sub(IMAGE_PLACEHOLDER, image_token_se, qs)
        else:
            qs = re.sub(IMAGE_PLACEHOLDER, DEFAULT_IMAGE_TOKEN, qs)
    else:
        if model.config.mm_use_im_start_end:
            qs = image_token_se + "\n" + qs
        else:
            qs = DEFAULT_IMAGE_TOKEN + "\n" + qs

    # Auto-detect conversation mode
    conv = conv_templates[args.conv_mode].copy()
    conv.append_message(conv.roles[0], qs)
    conv.append_message(conv.roles[1], None)
    prompt = conv.get_prompt()

    # Process image and generate
    images_tensor = process_images(images, image_processor, model.config)
    input_ids = tokenizer_image_token(prompt, tokenizer, IMAGE_TOKEN_INDEX, return_tensors="pt")

    with torch.inference_mode():
        output_ids = model.generate(
            input_ids, images=images_tensor, image_sizes=image_sizes,
            do_sample=True if args.temperature > 0 else False,
            temperature=args.temperature, top_p=args.top_p,
            num_beams=args.num_beams, max_new_tokens=args.max_new_tokens,
            use_cache=True,
        )

    outputs = tokenizer.batch_decode(output_ids, skip_special_tokens=True)[0].strip()
    print(outputs)

Import

from llava.eval.run_llava import eval_model

CLI Arguments

Argument	Type	Default	Required	Description
--model-path	str	"facebook/opt-350m"	Yes	Model checkpoint path or HuggingFace model ID
--model-base	str	None	No*	Base model (required for unmerged LoRA)
--image-file	str	--	Yes	Image file path or URL
--query	str	--	Yes	Text question about the image
--conv-mode	str	None	No	Conversation template (auto-detected)
--sep	str	","	No	Separator for multiple image paths
--temperature	float	0.2	No	Sampling temperature (0 for greedy)
--top_p	float	None	No	Top-p nucleus sampling
--num_beams	int	1	No	Beam search width
--max_new_tokens	int	512	No	Maximum tokens to generate

Inputs

• Model path -- Path to a merged model checkpoint, HuggingFace model ID, or LoRA adapter directory
• Image file -- Local image path or HTTP/HTTPS URL (PIL-compatible formats: JPEG, PNG, etc.)
• Text query -- Natural language question about the image
• Generation parameters -- Temperature, top_p, num_beams, max_new_tokens

Outputs

Generated text response printed to stdout. The output is the model's response to the visual question, decoded with special tokens stripped.

CLI Usage

Merged Model Inference

python -m llava.eval.run_llava \
    --model-path /path/to/merged-model \
    --image-file test.jpg \
    --query "Describe this image in detail."

Unmerged LoRA Model Inference

python -m llava.eval.run_llava \
    --model-path /path/to/lora-adapter \
    --model-base liuhaotian/llava-v1.5-13b \
    --image-file test.jpg \
    --query "What objects are visible in this image?"

Inference from URL with Custom Parameters

python -m llava.eval.run_llava \
    --model-path liuhaotian/llava-v1.5-13b \
    --image-file "https://example.com/photo.jpg" \
    --query "What is happening in this scene?" \
    --temperature 0 \
    --max_new_tokens 1024

Greedy Decoding for Reproducible Validation

python -m llava.eval.run_llava \
    --model-path ./checkpoints/llava-v1.5-13b-task-merged \
    --image-file validation_image.jpg \
    --query "Describe this image." \
    --temperature 0 \
    --max_new_tokens 256

Metadata

Field	Value
last_updated	2026-02-13 14:00 GMT
source_repo	Haotian_liu_LLaVA
commit	799f5f207c89
type	Implementation (API Doc)

Related Pages

• implements Principle:Haotian_liu_LLaVA_Model_Validation