Principle:EvolvingLMMs Lab Lmms eval HTTP API Model Serving
| Knowledge Sources | |
|---|---|
| Domains | Model Deployment, Distributed Systems |
| Last Updated | 2026-02-14 00:00 GMT |
Overview
HTTP API model serving decouples model inference from evaluation by exposing models through standardized REST endpoints.
Description
HTTP API model serving enables model inference to run as a separate service accessible via HTTP requests. This architecture separates the evaluation harness from the model execution environment, allowing evaluations to run on different machines, bypass complex dependency requirements (like Docker or specialized hardware drivers), and scale horizontally by running multiple server instances. The pattern uses base64-encoded media transmission, JSON request/response format, retry logic for robustness, and async/await for concurrent request handling.
Usage
Apply this principle when the model requires specialized hardware or runtime environments (e.g., TT-NN, TPU), you need to evaluate models without installing heavy dependencies locally, you want to scale inference across multiple server instances, or you're running evaluations in restricted environments (e.g., CI/CD pipelines).
Theoretical Basis
Architecture Components
- Server: Hosts the model and exposes inference endpoints (e.g., /audio/transcriptions)
- Client: Evaluation harness sends requests to server and processes responses
- Transport: HTTP/HTTPS with JSON payloads and base64-encoded media
- Authentication: Bearer token or API key for access control
- Retry Logic: Exponential backoff for transient failures
Request Flow
- Encode: Convert media (audio/image/video) to base64 for transmission
- Request: POST to endpoint with JSON payload containing encoded media
- Process: Server decodes media, runs model inference, formats result
- Response: Server returns JSON with prediction/transcription
- Parse: Client extracts result from response and continues evaluation
Benefits
- Decoupling: Evaluation code independent of model runtime
- Scalability: Multiple servers handle concurrent requests
- Flexibility: Swap model versions without changing evaluation code
- Isolation: Model dependencies isolated in server container
- Portability: Evaluation runs anywhere with network access
Implementation Patterns
- Synchronous: Sequential requests with retry logic (simple, lower throughput)
- Asynchronous: Parallel requests with asyncio/aiohttp (complex, higher throughput)
- Batch Endpoints: Server processes multiple samples in single request (most efficient)
Error Handling
- Network timeouts: Configurable timeout parameter (e.g., 300 seconds)
- Transient failures: Retry with exponential backoff (e.g., 3 retries)
- Malformed responses: Parse with fallback to string representation
- Server errors: Log and optionally return empty/default response