Implementation:EvolvingLMMs Lab Lmms eval Launch Server

Knowledge Sources	lmms-eval
Domains	Server, Infrastructure
Last Updated	2026-02-14 00:00 GMT

Overview

Concrete tool for starting an HTTP evaluation server with configurable host, port, and job management settings provided by the lmms-eval framework.

Description

The launch_server function is the main entry point for starting the lmms-eval HTTP evaluation server. It accepts a ServerArgs dataclass that encapsulates all server configuration, stores it on the FastAPI application state so that the lifespan handler can access it, and then delegates to Uvicorn to bind the ASGI application to the specified host and port. The function is blocking and does not return until the server is shut down.

The ServerArgs dataclass validates its fields at construction time: port must be an integer between 1 and 65535, and max_completed_jobs must be a positive integer. It also provides from_dict and to_dict convenience methods for serialization.

The FastAPI application is initialized with a lifespan context manager that creates a JobScheduler on startup (using max_completed_jobs and temp_dir_prefix from the server args) and stops it on shutdown.

Usage

Use this implementation when you need to:

Start the evaluation server programmatically from Python code
Configure server networking and job management parameters
Launch a standalone evaluation service from the command line

Code Reference

Source Location

Repository: lmms-eval
File: lmms_eval/entrypoints/http_server.py
Lines: L242-265 (launch_server), L44-70 (lifespan), L73-78 (app)
File: lmms_eval/entrypoints/server_args.py
Lines: L4-37 (ServerArgs)

Signature

def launch_server(args: ServerArgs) -> None:
    """
    Launch the evaluation server with the given arguments.

    Args:
        args: ServerArgs instance containing host, port, and scheduler configuration.
    """

@dataclass
class ServerArgs:
    """Arguments for launching the evaluation server."""

    host: str = field(default="localhost")
    port: int = field(default=8000)
    max_completed_jobs: int = field(default=100)
    temp_dir_prefix: str = field(default="lmms_eval_")

Import

from lmms_eval.entrypoints.http_server import launch_server
from lmms_eval.entrypoints.server_args import ServerArgs

I/O Contract

Inputs

Name	Type	Required	Description
args	`ServerArgs`	Yes	Configuration dataclass for the server
args.host	`str`	No (default: "localhost")	Network interface to bind to. Use "0.0.0.0" for all interfaces.
args.port	`int`	No (default: 8000)	TCP port to listen on. Validated to be between 1 and 65535.
args.max_completed_jobs	`int`	No (default: 100)	Maximum number of completed/failed/cancelled jobs to retain in memory before cleanup.
args.temp_dir_prefix	`str`	No (default: "lmms_eval_")	Prefix for temporary output directories created by the job scheduler.

Outputs

Name	Type	Description
(return)	`None`	Function blocks until server shutdown; does not return a value.
Side effect	Running FastAPI server	HTTP server bound to the specified host and port, accepting evaluation requests.
Side effect	JobScheduler	Background task processing evaluation jobs sequentially from the queue.

Usage Examples

Basic Example

from lmms_eval.entrypoints.http_server import launch_server
from lmms_eval.entrypoints.server_args import ServerArgs

# Launch with default settings (localhost:8000)
args = ServerArgs()
launch_server(args)

Custom Configuration Example

from lmms_eval.entrypoints.http_server import launch_server
from lmms_eval.entrypoints.server_args import ServerArgs

# Launch on all interfaces, custom port, retain more completed jobs
args = ServerArgs(
    host="0.0.0.0",
    port=8080,
    max_completed_jobs=200,
    temp_dir_prefix="my_eval_",
)
launch_server(args)

Command Line Example

python -m lmms_eval.launch_server --host 0.0.0.0 --port 8000

From Dictionary Configuration

from lmms_eval.entrypoints.http_server import launch_server
from lmms_eval.entrypoints.server_args import ServerArgs

config = {"host": "0.0.0.0", "port": 9000, "max_completed_jobs": 50}
args = ServerArgs.from_dict(config)
launch_server(args)

Related Pages

Implements Principle

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment