Workflow:Vllm project Vllm OpenAI Compatible Serving
| Knowledge Sources | |
|---|---|
| Domains | LLMs, Inference, API_Serving, MLOps |
| Last Updated | 2026-02-08 13:00 GMT |
Overview
End-to-end process for deploying a Large Language Model as an OpenAI-compatible HTTP API server using vLLM's built-in serving infrastructure.
Description
This workflow covers launching vLLM as a production-grade API server that implements the OpenAI Chat Completions and Completions API protocols. It supports streaming responses, tool/function calling, structured outputs, and concurrent request handling. The server uses Uvicorn/FastAPI under the hood and exposes endpoints compatible with the OpenAI Python SDK, enabling drop-in replacement for OpenAI API calls with local or self-hosted models.
Usage
Execute this workflow when you need to serve an LLM as a persistent HTTP endpoint for real-time inference. Typical scenarios include building chatbot backends, integrating LLM capabilities into applications via REST APIs, replacing OpenAI API calls with self-hosted models, and serving models behind load balancers in production deployments.
Execution Steps
Step 1: Install vLLM
Install the vLLM package with all serving dependencies. The installation includes FastAPI, Uvicorn, and the OpenAI-compatible API layer in addition to the core inference engine.
Key considerations:
- The serving layer is included in the default vLLM installation
- For TLS/SSL support, additional certificate configuration is needed
- Production deployments may benefit from running behind a reverse proxy
Step 2: Select and Configure Model
Choose a HuggingFace model and determine the serving configuration including tensor parallelism, quantization, context length limits, and GPU memory allocation. These settings directly impact throughput, latency, and concurrent request capacity.
Key considerations:
- Model selection determines memory requirements and supported features
- Quantized models (GPTQ, AWQ, FP8) reduce memory footprint
- tensor_parallel_size should match available GPUs
- max_model_len caps the maximum context window served
- gpu_memory_utilization balances KV cache size against safety margin
Step 3: Launch the API Server
Start the vLLM server using the CLI command with the chosen model and configuration. The server initializes the engine, loads model weights, and begins listening for HTTP requests on the specified host and port.
Pseudocode:
# Launch server via CLI # vllm serve <model_name> --host 0.0.0.0 --port 8000 [options] # Server starts Uvicorn with FastAPI application # Engine initializes model, allocates KV cache # Server ready to accept requests
Key considerations:
- Default port is 8000; configurable via --port
- Use --host 0.0.0.0 to accept external connections
- --chat-template can override the model's default template
- --api-key enables API key authentication
- --served-model-name customizes the model name in API responses
Step 4: Send Requests via OpenAI Client
Connect to the server using the OpenAI Python SDK or any HTTP client. The server supports the /v1/chat/completions, /v1/completions, and /v1/embeddings endpoints with standard OpenAI request formats.
Key considerations:
- Set base_url to point to the vLLM server address
- Streaming responses are supported via stream=True
- Tool/function calling follows the OpenAI tool_choice protocol
- Structured outputs can be requested via response_format parameter
Step 5: Handle Streaming Responses
For real-time applications, consume server-sent events (SSE) from the streaming endpoint. Each chunk contains a partial response delta that can be displayed incrementally to users.
Key considerations:
- Streaming reduces time-to-first-token perception
- Each SSE chunk follows the OpenAI delta format
- Connection should handle reconnection for long-running streams
- Non-streaming mode returns the complete response in one JSON payload
Step 6: Monitor and Scale
Monitor server health, throughput, and latency using the built-in metrics endpoint. vLLM exposes Prometheus-compatible metrics for integration with Grafana dashboards and alerting systems.
Key considerations:
- /metrics endpoint provides Prometheus-format statistics
- Key metrics: request throughput, token generation rate, queue depth
- Horizontal scaling via multiple server instances behind a load balancer
- Grafana dashboard templates are provided in the examples directory