Principle:EvolvingLMMs Lab Lmms eval MCP Tool Integration

Knowledge Sources	EvolvingLMMs_Lab_Lmms_eval
Domains	Tool Integration, Model Context Protocol
Last Updated	2026-02-14 00:00 GMT

Overview

MCP (Model Context Protocol) Tool Integration enables language models to access external tools and resources through the Model Context Protocol. This principle establishes how the framework implements MCP servers that expose tools for video processing, image manipulation, and other utilities that can be invoked by language models during evaluation.

Theoretical Basis

MCP Server Definition

MCP servers are FastMCP-based applications that expose tools through a standardized protocol:

• Define tools using the @app.tool() decorator with name and description
• Tools accept typed parameters with Pydantic Field annotations
• Tools return structured content (ImageContent, TextContent, AudioContent)
• Servers run as standalone processes that can be invoked by MCP clients

Tool Registration

Tools are registered with the MCP server using decorators:

• Each tool has a unique name and description
• Input parameters are annotated with types and field descriptions
• Return types specify the format of tool outputs
• Parameter validation is built into the tool definition

Content Types

MCP tools return standardized content types:

• ImageContent: Base64-encoded images with MIME type
• TextContent: Plain text responses
• AudioContent: Base64-encoded audio with MIME type
• Tools can return single items or lists of content

Client-Server Communication

MCP clients communicate with servers through stdio:

• Servers are launched as subprocesses with Python command
• ClientSession manages bidirectional communication streams
• Async/await pattern for non-blocking tool execution
• Timeout configuration for long-running operations

Design Patterns

Server Implementation

• FastMCP Application: Creates the MCP server instance with name and version
• Tool Functions: Decorated functions that implement tool logic
• Parameter Validation: Input validation with detailed error messages
• Content Encoding: Converting outputs to MCP-compatible formats

Client Implementation

• MCPClient Class: Manages connections to MCP servers
• Function Discovery: Lists available tools and their schemas
• Tool Execution: Invokes tools with arguments and retrieves results
• Format Conversion: Converts MCP responses to OpenAI-compatible format

Error Handling

• Parameter validation with specific error messages
• File existence checks for resource-based tools
• Exception wrapping with context information
• Logging for debugging server operations

Tool Design

• Keep tools focused on single responsibilities
• Provide clear parameter descriptions for model consumption
• Return structured content that models can interpret
• Validate all inputs before processing

Performance

• Use async operations for I/O-bound tasks
• Configure appropriate timeouts for tool execution
• Consider caching for expensive operations
• Stream large results when possible

Integration

• Tools should be stateless when possible
• Use standard content types for interoperability
• Document tool behavior and requirements
• Provide example usage for each tool

Usage Examples

MCP servers are launched independently and accessed by models:

# Server side (crop_video_mcp_server.py)
app = FastMCP("Video Tools MCP Server", "0.1.0")

@app.tool(name="crop_video")
def crop_video(video_path: str, start_time: float, end_time: float):
    # Process video and return frames as ImageContent
    ...

# Client side
client = MCPClient("/path/to/server.py")
functions = client.get_function_list_sync()
result = client.run_tool_sync("crop_video", {"video_path": "...", ...})

Best Practices

• Always validate tool parameters before processing
• Use descriptive tool and parameter names for LLM understanding
• Return structured content with proper MIME types
• Log operations for debugging and monitoring
• Handle errors gracefully with informative messages
• Document expected parameter formats and ranges
• Test tools independently before integration

Related Pages

Implementations

• EvolvingLMMs_Lab_Lmms_eval_Crop_Video_MCP_Server — video cropping and frame extraction tool
• EvolvingLMMs_Lab_Lmms_eval_Sample_MCP_Server — example MCP tools for image and weather queries
• Implementation:EvolvingLMMs_Lab_Lmms_eval_MCP_Client

See Also

• EvolvingLMMs_Lab_Lmms_eval_Model_Inference — MCP tools extend model capabilities during inference
• EvolvingLMMs_Lab_Lmms_eval_Request_Construction — tool results can be incorporated into model requests
• EvolvingLMMs_Lab_Lmms_eval_Media_Handling — MCP tools often process images and videos