Principle:EvolvingLMMs Lab Lmms eval TUI Server Architecture

Knowledge Sources	EvolvingLMMs_Lab_Lmms_eval
Domains	Web_UI, Evaluation, Backend
Last Updated	2026-02-14 00:00 GMT

Overview

TUI Server Architecture refers to the design pattern used to implement the web-based Terminal User Interface (TUI) backend for LMMs-Eval. This architecture provides a RESTful API layer that manages evaluation job lifecycle, streams real-time output, and serves the frontend application.

Theoretical Basis

Core Components

The TUI Server Architecture consists of several key components:

API Layer: A FastAPI-based REST API that exposes endpoints for:

Health checks and system information
Model and task discovery
Command preview generation
Evaluation job management (start, stop, stream)

Job Management: In-memory job storage that tracks:

Job status (starting, running, completed, error, stopped)
Associated subprocess handles
User configuration and commands

Process Execution: Asynchronous subprocess management using asyncio:

Shell command construction from user configuration
Environment variable injection
Process group management for clean termination

Output Streaming: Server-Sent Events (SSE) for real-time log streaming:

Non-blocking output capture from subprocess stdout/stderr
Event-driven architecture with typed messages (output, done, stopped, error)
Connection lifecycle management

Static File Serving: Integrated web server for SPA hosting:

Serves compiled React frontend from dist directory
Fallback routing for client-side navigation

Design Patterns

Separation of Concerns: The server cleanly separates:

API endpoints from business logic
Command building from execution
Configuration parsing from subprocess management

Stateless API with Stateful Jobs: REST endpoints are stateless, while job state is maintained in-memory with UUID-based tracking.

Event Streaming: Uses SSE for unidirectional server-to-client streaming, avoiding polling overhead and providing low-latency updates.

Process Isolation: Uses process groups (start_new_session=True) to ensure child processes can be cleanly terminated without orphaning subprocesses.

Command Construction

The architecture employs a dual-command pattern:

Display Command: Human-readable format with line breaks and export statements for UI preview Execution Command: Shell-compatible format with proper quoting and chained exports for subprocess execution

This separation ensures the preview accurately reflects what will run while maintaining shell compatibility.

Security Considerations

The architecture includes several security-relevant design choices:

CORS middleware configured for development (* origin)
No authentication layer (intended for local/trusted environments)
Direct shell command execution (subprocess.shell=True)
Process group termination via SIGTERM (graceful shutdown)

Integration Points

Discovery System

The server integrates with the discovery cache to dynamically enumerate available models and tasks:

Uses get_discovery_cache() to access registered models/tasks
Supports include_all parameter for comprehensive listings
Task groups identified by name prefix "[Group]"

Subprocess Management

Key subprocess patterns:

asyncio.create_subprocess_shell for command execution
stdout=PIPE with stderr=STDOUT for unified output
start_new_session=True for process group creation
os.killpg for group termination

Frontend Integration

The server serves as a backend for the React frontend:

Provides JSON APIs for configuration
Streams output via SSE endpoint
Serves compiled static assets from dist directory
Implements SPA fallback routing

Configuration Model

The architecture uses Pydantic models for type-safe configuration:

class EvalRequest(BaseModel):
    model: str
    model_args: str = ""
    tasks: list[str]
    env_vars: str = ""
    batch_size: int = 1
    limit: int | None = 10
    output_path: str = "./logs/"
    log_samples: bool = True
    verbosity: str = "INFO"
    device: str | None = None

This ensures validation at the API boundary and provides automatic OpenAPI documentation.

Environment Variable Handling

The server normalizes environment variable syntax:

Strips comments and empty lines
Adds "export" prefix if missing
Preserves existing export statements
Builds shell-compatible export chains

This allows flexible input formats while ensuring correct shell execution.

System Information Exposure

The health endpoint provides runtime context:

Package version from metadata
Git branch and commit (if available)
Hostname, platform, Python version
Current working directory and repository root

This information aids debugging and ensures users know what version is running.

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment