Workflow:Haotian liu LLaVA Web Demo Deployment
| Knowledge Sources | |
|---|---|
| Domains | LLMs, Serving, Multimodal, Deployment |
| Last Updated | 2026-02-13 23:00 GMT |
Overview
Deploy a LLaVA model as an interactive web demo using the distributed controller-worker-frontend architecture with Gradio.
Description
This workflow sets up LLaVA's three-tier serving infrastructure for interactive multimodal chat. The architecture consists of a central controller that manages model workers, one or more model workers that handle inference, and a Gradio web frontend that provides a chat interface with image upload capabilities.
The system supports multiple concurrent model workers (enabling A/B comparison between checkpoints), streaming text generation, LoRA models with separate base model specification, and optional 4-bit/8-bit quantization for reduced GPU memory. An alternative SGLang backend is available for higher throughput inference.
Usage
Execute this workflow when you want to interactively test a LLaVA model checkpoint through a web interface, demonstrate the model to stakeholders, or run a persistent inference service for multimodal chat.
Execution Steps
Step 1: Launch Controller
Start the central controller service which manages worker registration, heartbeat monitoring, and request dispatching. The controller maintains a registry of active model workers and routes client requests using either lottery or shortest-queue dispatch strategies.
Key considerations:
- The controller runs as a FastAPI server (default port 10000)
- Workers must be able to reach the controller address for registration and heartbeats
- Worker heartbeats expire after a configurable timeout (default: expiration period in constants)
Step 2: Launch Model Worker
Start one or more model worker processes, each loading a LLaVA checkpoint and registering with the controller. The worker loads the model using the unified model builder, initializes the vision tower and projector, and exposes inference endpoints for streaming text generation.
Key considerations:
- Each worker runs on a separate port and registers with the controller
- For LoRA models, provide --model-base pointing to the base language model
- Use --load-4bit or --load-8bit for quantized inference to reduce VRAM
- Multiple workers can serve different model checkpoints for comparison
- Wait for the Uvicorn running on... message before proceeding
Step 3: Launch Gradio Web Interface
Start the Gradio web server which provides the interactive chat UI. The frontend connects to the controller to discover available models and routes user queries to the appropriate model worker. It supports image upload, multi-turn conversation, and streaming text output.
Key considerations:
- The web server queries the controller's model list and displays available models
- Use --model-list-mode reload to dynamically refresh available models
- Image processing (resize, padding, aspect ratio handling) happens on the worker side
- Conversation logs are saved for analysis
Step 4: Alternative CLI Interface
For terminal-based interaction without the full distributed setup, use the CLI chat interface. This provides a single-process interactive session with image input and streaming text output directly in the terminal.
Key considerations:
- The CLI loads the model directly without needing the controller/worker infrastructure
- Supports the same --model-base, --load-4bit, and --load-8bit options
- Image is loaded once at the start and used across multiple conversation turns
- Useful for quick testing before deploying the full web demo