Principle:EvolvingLMMs Lab Lmms eval TUI Command Interface

Overview

The TUI Command Interface principle governs the implementation of the text-based user interface (TUI) and web UI for lmms_eval. It encompasses CLI launcher functionality, resource discovery mechanisms, and web server integration to provide an accessible interface for configuring and running evaluations.

Core Concepts

Web UI Integration

The TUI provides a browser-based interface that:

• Automatically builds the web UI on first launch
• Manages the FastAPI server lifecycle
• Opens the browser automatically to the correct URL
• Handles graceful shutdown on interrupt

Resource Discovery

Automatic discovery of available resources enables:

• Dynamic task listing from YAML configurations
• Model registry enumeration
• Curated "popular" lists for quick access
• Caching for performance optimization

User Experience Focus

The interface prioritizes ease of use:

• No configuration files required
• Browser auto-launch for immediate access
• Progress indication during build
• Clear error messages

Design Principles

Progressive Disclosure

The interface uses a two-tier approach:

• Default view: Shows curated popular tasks and models
• Full view: Discovers and displays all available options on demand

This reduces cognitive load for new users while maintaining full power for advanced use cases.

Lazy Initialization

Resources are discovered and cached only when needed, improving startup time and responsiveness.

Environment-Based Configuration

Server settings are controlled through environment variables rather than command-line arguments, providing flexibility without complexity.

Graceful Degradation

Discovery failures don't break the interface - empty lists are returned and the UI remains functional.

Self-Contained Build System

The web UI build is managed entirely within the CLI:

• Checks for existing builds
• Installs dependencies automatically
• Provides clear progress indication
• Fails gracefully with helpful errors

Implementation Guidelines

CLI Launcher

The launcher should:

• Check for and build the web UI if needed
• Start the server with appropriate configuration
• Open the browser automatically
• Handle shutdown signals cleanly with timeout-based fallback

Discovery System

Discovery implementations should:

• Scan relevant directories recursively
• Parse configuration files safely
• Handle errors without crashing
• Generate human-friendly display names
• Cache results for performance

Web Server

The server should:

• Expose a health endpoint for readiness checks
• Run on a configurable port
• Bind to all interfaces (0.0.0.0) for flexibility
• Integrate with FastAPI for modern async support

Caching Strategy

Cache implementations should:

• Separate popular and full discovery results
• Support reloading for detecting changes
• Provide cache invalidation
• Use lazy loading for initial population

Key Components

CLI Module

Responsibilities:

• Web UI build orchestration
• Server process management
• Browser launching
• Shutdown handling

Discovery Module

Responsibilities:

• Task YAML scanning and parsing
• Model registry enumeration
• Display name generation
• Result caching

Cache Class

Responsibilities:

• Lazy initialization of discovery results
• Separate storage for popular vs. all items
• Reload and clear operations
• Thread-safe access patterns (if needed)

Usage Patterns

Starting the TUI

# Default port (8000)
python -m lmms_eval.tui.cli

# Custom port
LMMS_SERVER_PORT=3000 python -m lmms_eval.tui.cli

Discovery in Server

from lmms_eval.tui.discovery import get_discovery_cache

cache = get_discovery_cache()

# Get popular tasks
popular_tasks = cache.get_tasks(include_all=False)

# Get all tasks
all_tasks = cache.get_tasks(include_all=True)

# Reload after changes
task_count, model_count = cache.reload()

Error Handling

Build Failures

• Missing package.json: Report source not found
• npm install failure: Report dependency installation failure
• npm build failure: Report build failure
• All return non-zero exit code for scriptability

Server Failures

• Port in use: Let uvicorn handle the error message
• Server crash: Wait for subprocess to complete
• Keyboard interrupt: Clean shutdown with timeout

Discovery Failures

• Missing directories: Silently skip
• Invalid YAML: Silently skip individual files
• Import errors: Return empty lists

Best Practices

For CLI Implementation

• Always check prerequisites before starting
• Provide clear progress indication
• Use subprocess for isolation
• Handle signals for clean shutdown
• Preserve user's working directory

For Discovery Implementation

• Use safe YAML loading
• Ignore known non-task directories
• Generate display names consistently
• Cache aggressively
• Handle missing fields gracefully

For Cache Management

• Initialize lazily
• Separate concerns (popular vs. all)
• Provide clear reload semantics
• Consider thread safety for concurrent access

Related Principles

• Task_Directory_Structure: Task organization for discovery
• YAML_Task_Configuration: Task config format
• Registry_Registration: Model registration

Implementations

• TUI_CLI: CLI launcher for web UI
• TUI_Discovery: Resource discovery system
• Implementation:EvolvingLMMs_Lab_Lmms_eval_TUI_CLI