Principle:EvolvingLMMs Lab Lmms eval Dependency Management

Knowledge Sources	EvolvingLMMs_Lab_Lmms_eval
Domains	Package Management, Import Handling
Last Updated	2026-02-14 00:00 GMT

Overview

Dependency Management provides a unified system for handling optional dependencies in the evaluation framework. This principle establishes how the framework checks for, imports, and requires optional packages while providing clear error messages when dependencies are missing.

Theoretical Basis

Optional Dependencies

The framework has a core set of required dependencies and many optional dependencies:

• Core Dependencies: Always installed (numpy, torch, transformers, etc.)
• Optional Dependencies: Installed via extras (decord, textual, accelerate, etc.)
• Feature-Specific: Certain features require specific packages
• Graceful Degradation: Missing optional deps should not crash unrelated features

Import Strategies

Eager Import

import torch  # Fails immediately if missing

Used for core dependencies that are always required.

Optional Import

VideoReader, has_decord = optional_import("decord", "VideoReader")
if has_decord:
    # Use VideoReader
else:
    # Fallback or skip feature

Used for optional dependencies with runtime checking.

Lazy Import

# Module-level lazy loading
__getattr__ = make_lazy_getattr({
    "run_tui": ("lmms_eval.tui.cli", "main"),
})

Delays import until first access.

Dependency Checking

Multiple levels of checking:

• Availability Check: Is package installed?
• Version Check: Does version meet requirements?
• Capability Check: Does package support needed features?
• Runtime Check: Check when feature is used

Error Communication

Clear, actionable error messages:

• Explain what is missing
• Explain why it is needed
• Provide installation command
• Reference extras group if applicable

Design Patterns

Package Availability

• is_package_available(): Cached check for package installation
• importlib.util.find_spec(): Standard library method for checking
• LRU Cache: Avoid repeated filesystem checks

Optional Import

• optional_import(): Import with fallback value
• Tuple Return: (imported_object, is_available)
• Attribute Access: Can import specific attributes from modules

Required Dependencies

• require_package(): Enforce dependency presence
• MissingOptionalDependencyError: Custom exception with helpful message
• Extras Groups: Map packages to installation groups

Lazy Loading

• make_lazy_getattr(): Create module-level __getattr__
• Deferred Import: Import only when accessed
• AttributeError: Raise for undefined attributes

Performance

• Cache availability checks to avoid repeated filesystem operations
• Lazy loading for heavyweight imports
• Minimize import time for CLI startup
• Only import what is needed for current operation

User Experience

• Clear error messages with installation instructions
• Graceful degradation when optional features unavailable
• No cryptic ImportError messages
• Document which features require which dependencies

Maintainability

• Central location for import utilities
• Consistent patterns across codebase
• Easy to add new optional dependencies
• Clear mapping of features to dependencies

Robustness

• Handle missing packages gracefully
• Validate package capabilities when needed
• Support multiple installation methods
• Consider version compatibility

Usage Examples

Optional Import with Fallback

from lmms_eval.imports import optional_import

# Import with fallback
VideoReader, has_decord = optional_import("decord", "VideoReader")

if has_decord:
    video = VideoReader(path)
else:
    print("Video processing unavailable: install with pip install lmms_eval[video]")

Checking Package Availability

from lmms_eval.imports import is_package_available

if is_package_available("textual"):
    from textual.app import App
    # Use TUI
else:
    print("TUI not available. Install with: pip install lmms_eval[tui]")

Requiring Dependencies

from lmms_eval.imports import require_package

def run_tui():
    require_package("textual", extras="tui", feature="TUI interface")
    # If we get here, package is installed
    from textual.app import App
    # ... use TUI

Lazy Module Loading

# In module __init__.py
from lmms_eval.imports import make_lazy_getattr

__getattr__ = make_lazy_getattr({
    "run_tui": ("lmms_eval.tui.cli", "main"),
    "VideoModel": ("lmms_eval.models.video", "VideoModel"),
})

# In user code
from lmms_eval import run_tui  # Only imports when accessed

Configuration

pyproject.toml Extras

[project.optional-dependencies]
video = ["decord>=0.6.0", "opencv-python"]
tui = ["textual>=0.40.0"]
api = ["fastapi", "uvicorn"]
all = ["lmms_eval[video,tui,api]"]

Import Mapping

EXTRAS_MAP = {
    "decord": "video",
    "textual": "tui",
    "fastapi": "api",
    # ... more mappings
}

Error Message Examples

Missing Package with Extras

'textual' is required for TUI interface but not installed.
Install with: pip install lmms_eval[tui]

Missing Package without Extras

'custom_package' is required but not installed.
Install with: pip install custom_package

Missing Package for Feature

'decord' is required for video processing but not installed.
Install with: pip install lmms_eval[video]

Best Practices

For Framework Developers

• Use optional_import for all optional dependencies
• Provide clear fallback behavior
• Map packages to extras groups
• Document which features need which packages
• Test with minimal dependencies installed
• Cache availability checks when used repeatedly
• Use lazy loading for heavyweight imports

For Model Developers

• Declare model-specific dependencies clearly
• Check for dependencies in model __init__
• Provide helpful error messages
• Consider fallback implementations
• Document installation requirements

For Task Developers

• Check for task-specific dependencies
• Provide alternative implementations when possible
• Skip tests if dependencies unavailable
• Document minimum dependency versions

Common Patterns

Video Processing

from lmms_eval.imports import optional_import, require_package

VideoReader, has_decord = optional_import("decord", "VideoReader")

def load_video(path):
    if not has_decord:
        require_package("decord", extras="video", feature="video processing")

    return VideoReader(path)

Model-Specific Imports

class CustomModel(BaseModel):
    def __init__(self):
        require_package("custom_lib", feature=f"{self.model_name} model")
        from custom_lib import Model
        self.model = Model()

Feature Detection

from lmms_eval.imports import is_package_available

FEATURES = {
    "video": is_package_available("decord"),
    "tui": is_package_available("textual"),
    "api": is_package_available("fastapi"),
}

def list_features():
    for feature, available in FEATURES.items():
        status = "available" if available else "not available"
        print(f"{feature}: {status}")

Conditional Imports

from lmms_eval.imports import optional_import

# Try multiple libraries
Image, has_pil = optional_import("PIL", "Image")
if not has_pil:
    Image, has_pillow = optional_import("pillow", "Image")

if not (has_pil or has_pillow):
    raise ImportError("Either PIL or Pillow required")

Integration Points

CLI Entry Point

def main():
    # Check for required packages
    if args.use_tui:
        require_package("textual", extras="tui")

    # Import based on availability
    if is_package_available("accelerate"):
        use_accelerate = True
    else:
        use_accelerate = False

Model Registry

def get_model(model_type):
    if model_type == "video_model":
        require_package("decord", extras="video")

    # Import and return model
    return MODEL_REGISTRY[model_type]()

Task Loading

def load_task(task_name):
    task = TASK_REGISTRY[task_name]

    # Check task dependencies
    if hasattr(task, "required_packages"):
        for pkg, extras in task.required_packages.items():
            require_package(pkg, extras=extras)

    return task()

Testing Considerations

Mock Missing Dependencies

def test_missing_dependency(monkeypatch):
    monkeypatch.setattr("lmms_eval.imports.is_package_available", lambda x: False)

    with pytest.raises(MissingOptionalDependencyError):
        require_package("some_package")

Test Optional Features

@pytest.mark.skipif(not is_package_available("decord"), reason="decord not installed")
def test_video_processing():
    # Test video features
    pass

Test Import Patterns

def test_optional_import():
    obj, available = optional_import("nonexistent_package")
    assert available is False
    assert obj is None

Future Enhancements

• Version checking and compatibility
• Automatic dependency installation prompts
• Dependency graph visualization
• Bundle size optimization
• Lazy submodule loading
• Dynamic feature registration based on available deps
• Dependency conflict detection

Related Pages

Implementations

• EvolvingLMMs_Lab_Lmms_eval_Import_Utils — core import utilities
• Implementation:EvolvingLMMs_Lab_Lmms_eval_Import_Utils

See Also

• EvolvingLMMs_Lab_Lmms_eval_Model_Type_Selection — models may require specific dependencies
• EvolvingLMMs_Lab_Lmms_eval_Media_Handling — video/image processing requires optional packages
• EvolvingLMMs_Lab_Lmms_eval_Example_Script_Creation — examples demonstrate optional features