Implementation:OpenGVLab InternVL LLaVA Controller

Knowledge Sources	OpenGVLab_InternVL
Domains	Serving, Distributed Systems, LLaVA
Last Updated	2026-02-07 14:00 GMT

Overview

Distributed worker management controller for the LLaVA serving infrastructure, implemented as a FastAPI server that tracks model workers via heartbeats and dispatches client requests using configurable load-balancing strategies.

Description

The Controller class maintains a registry of WorkerInfo dataclass instances (tracking model_names, speed, queue_length, check_heart_beat, and last_heart_beat) and supports two DispatchMethod strategies: LOTTERY (weighted random selection based on worker speed) and SHORTEST_QUEUE (routes to the worker with the lowest queue-to-speed ratio). A background thread periodically checks heartbeats and removes workers that exceed the CONTROLLER_HEART_BEAT_EXPIRATION timeout. The controller exposes REST endpoints via FastAPI: /register_worker for worker registration, /refresh_all_workers to re-validate all workers, /list_models to aggregate available model names, /get_worker_address to dispatch requests, /receive_heart_beat for worker liveness updates, /worker_generate_stream to proxy streaming generation requests, and /worker_get_status to provide aggregated status (enabling the controller to act as a hierarchical worker itself). The controller runs on configurable host/port (default localhost:21001).

Usage

Use this controller as the central coordination service when deploying multiple LLaVA model workers for distributed inference. Start it before any workers.

Code Reference

Source Location

• Repository: OpenGVLab_InternVL
• File: internvl_chat_llava/llava/serve/controller.py
• Lines: 1-298

Signature

class DispatchMethod(Enum):
    LOTTERY = auto()
    SHORTEST_QUEUE = auto()

@dataclasses.dataclass
class WorkerInfo:
    model_names: List[str]
    speed: int
    queue_length: int
    check_heart_beat: bool
    last_heart_beat: str

class Controller:
    def __init__(self, dispatch_method: str): ...
    def register_worker(self, worker_name, check_heart_beat, worker_status): ...
    def get_worker_address(self, model_name): ...
    def receive_heart_beat(self, worker_name, queue_length): ...
    def worker_api_generate_stream(self, params): ...
    def worker_api_get_status(self): ...

Import

# Standalone server script:
# python -m llava.serve.controller --host localhost --port 21001 --dispatch-method shortest_queue

I/O Contract

Inputs

Name	Type	Required	Description
--host	str	No	Server host (default: "localhost")
--port	int	No	Server port (default: 21001)
--dispatch-method	str	No	Load balancing strategy: "lottery" or "shortest_queue" (default: "shortest_queue")

Outputs

Name	Type	Description
REST API	FastAPI endpoints	Worker registration, model listing, request dispatch, and heartbeat management

Usage Examples

Basic Usage

# Start the controller server:
# python -m llava.serve.controller --port 21001 --dispatch-method shortest_queue

# Workers register via POST /register_worker
# Clients request worker addresses via POST /get_worker_address
# Streaming generation proxied via POST /worker_generate_stream

Related Pages

• Principle:OpenGVLab_InternVL_Distributed_Worker_Management