Implementation:Mit han lab Llm awq VILA15 Demo

Knowledge Sources	Mit_han_lab_Llm_awq
Domains	Demo, Multimodal
Last Updated	2026-02-15 00:00 GMT

Overview

Interactive multimodal chat demo for VILA 1.5 models with flash attention support, W4A16 AWQ quantization, chunk prefilling optimization for multi-turn conversations, and streaming text generation.

Description

This script provides an interactive command-line chat interface for VILA 1.5 vision-language models, extending the VILA 1.0 demo with chunk prefilling optimization and flash attention support.

image_parser splits the --image-file argument string on the separator (--im-sep, default comma) to produce a list of image file paths, identical in behavior to the VILA 1.0 version.

main handles the complete demo lifecycle:

Model Loading: The VilaLlamaForCausalLM model is loaded from AutoConfig with accelerated initialization (disabled parameter init functions). The tokenizer is loaded from the llm subdirectory of the model path and used to convert LLAVA_DEFAULT_IMAGE_PATCH_TOKEN to its token ID. The vision tower's image_processor is extracted. The model is cast to half precision.

Precision Modes: Two modes are supported. W16A16 loads the LLM component via load_checkpoint_and_dispatch from the llm subdirectory with module-level sharding, then moves the full model to the device. W4A16 loads AWQ-quantized weights for model.llm via load_awq_model (4-bit, group size 128) and applies fused kernel replacements. When --flash_attn is enabled, make_quant_attn is called with the flash attention flag set to 1; otherwise, standard attention is used. make_quant_norm is also applied.

Image Preparation: Images are loaded, optionally visualized, and preprocessed identically to the VILA 1.0 demo using load_images, vis_images, and process_images. The resulting tensor is placed on the target device in float16.

Chat Loop: An interactive loop manages user prompts and streaming generation via LlavaStreamGenerator. A key difference from VILA 1.0 is the --chunk_prefilling option: when enabled, start_pos tracks the cumulative token position across turns, and the image tensor is set to None after the first turn (freeing memory while retaining cached KV states). Each turn's total token count is returned by stream_output and accumulated in start_pos. Multi-turn conversation history is managed via model_prompter.update_template(outputs, args.chunk_prefilling), which accounts for the chunk prefilling mode. Empty input triggers exit with timing statistics from TimeStats.

Usage

Run from the command line to start an interactive VILA 1.5 chat session:

# W4A16 with flash attention and chunk prefilling
python tinychat/vila15_demo.py \
    --model-path /path/to/vila-1.5-7b \
    --quant-path /path/to/vila-1.5-7b-w4-g128-awq.pt \
    --precision W4A16 \
    --image-file image.jpg \
    --flash_attn \
    --chunk_prefilling

# Full precision inference
python tinychat/vila15_demo.py \
    --model-path /path/to/vila-1.5-7b \
    --precision W16A16 \
    --image-file https://example.com/image.jpg

Code Reference

Source Location

• Repository: Mit_han_lab_Llm_awq
• File: tinychat/vila15_demo.py
• Lines: 1-264

Signature

def image_parser(args):

def main(args):

Import

# CLI script, run directly:
python tinychat/vila15_demo.py [OPTIONS]

I/O Contract

image_parser

Parameter	Type	Description
args	Namespace	Parsed CLI args with image_file (str) and im_sep (str) attributes

Returns	Type	Description
image_paths	list[str]	List of image file paths split on separator

CLI Arguments

Argument	Type	Default	Description
--model_type	str	LLaMa	Model type for prompt template selection
--model-path	str	(required)	Path to VILA 1.5 model checkpoint
--quant-path	str	(path)	Path to AWQ quantized weight file
--precision	str	W4A16	Compute precision: "W16A16" or "W4A16"
--image-file	str	(URL)	Comma-separated image paths or URLs
--im-sep	str	,	Separator for multiple image paths
--device	str	cuda	CUDA device
--max_seq_len	int	2048	Maximum sequence length / KV cache size
--single_round	flag	False	Disable multi-turn conversation memory
--vis-image	flag	False	Visualize input images in terminal
--empty-prompt	flag	False	Use empty prompt template
--flash_attn	flag	False	Enable flash attention in quantized attention layers
--chunk_prefilling	flag	False	Enable chunk prefilling for multi-turn speedup

Interactive I/O

Direction	Description
Input	User text prompts via stdin; <image> placeholders supported; empty input exits
Output	Streamed assistant responses to stdout with timing statistics on exit

Usage Examples

# VILA 1.5 with flash attention, chunk prefilling, and terminal image display
python tinychat/vila15_demo.py \
    --model_type LLaMa \
    --model-path /models/vila-1.5-7b \
    --quant-path /models/vila-1.5-7b-w4-g128-awq.pt \
    --precision W4A16 \
    --image-file "photo.jpg" \
    --flash_attn \
    --chunk_prefilling \
    --vis-image \
    --max_seq_len 2048

# Multi-turn session with chunk prefilling:
# USER: What do you see in this image?
# ASSISTANT: I can see a landscape with mountains...
# (start_pos advances by total_tokens from first turn)
# USER: What season does it appear to be?
# ASSISTANT: Based on the foliage colors, it appears to be autumn...
# (start_pos advances again, no image reprocessing)
# USER: (empty to exit)
# EXIT... (timing stats: context time, generation speed, etc.)

Related Pages

• Principle:Mit_han_lab_Llm_awq_Interactive_Multimodal_Demo