Principle:Haotian liu LLaVA Model Validation
Overview
Quality assurance process for verifying that a fine-tuned vision-language model produces coherent visual reasoning outputs.
Description
Model validation after finetuning involves loading the trained model (merged or LoRA), providing test images with text queries, and verifying the outputs are coherent and relevant. This serves as a smoke test before deploying or running formal benchmark evaluation.
In LLaVA, validation uses the same inference pipeline as single-image inference:
- Load model via load_pretrained_model() (handles both merged and unmerged LoRA models)
- Preprocess the image through the CLIP vision tower's image processor
- Construct a conversation prompt using the appropriate template (auto-detected from model name)
- Tokenize the prompt with image token placeholders (IMAGE_TOKEN_INDEX = -200)
- Generate a response via model.generate()
- Decode and inspect the output text
For unmerged LoRA models, the --model-base argument must be specified to load the base model before applying adapters. For merged models, only --model-path is needed.
Usage
Use after LoRA training (with merged or unmerged weights) or full finetuning to verify the model works correctly before:
- Deploying to production via model_worker
- Running formal benchmark evaluation suites
- Sharing the model checkpoint publicly
Validation should cover:
- Basic image description tasks to verify coherent output
- Domain-specific queries matching the finetuning objective
- Edge cases (unusual images, complex questions) to test robustness
Theoretical Basis
Validation is a qualitative check. The model processes the image through the full CLIP vision encoder, multimodal projector, and LLM pipeline. The output is inspected for:
- Coherence -- The response forms grammatically correct, meaningful sentences
- Relevance -- The response addresses the visual content of the image
- Formatting -- The response follows expected output patterns
Using temperature=0.2 provides near-deterministic outputs for consistent testing. Setting temperature=0 enables fully greedy decoding, which is useful for reproducible validation results.
The conversation template is auto-detected based on the model name:
- "llava_llama_2" for LLaMA-2 based models
- "mistral_instruct" for Mistral-based models
- "llava_v1" for v1.x Vicuna-based models
- "mpt" for MPT-based models
Knowledge Sources
- Repo -- LLaVA -- https://github.com/haotian-liu/LLaVA
Domains
- Model_Evaluation
- Quality_Assurance
Metadata
| Field | Value |
|---|---|
| last_updated | 2026-02-13 14:00 GMT |
| source_repo | Haotian_liu_LLaVA |
| commit | 799f5f207c89 |
| type | Principle |