Implementation:EvolvingLMMs Lab Lmms eval Capability Prompt Templates

File: lmms_eval/tasks/capability/prompt.py (263 lines)

Overview

Comprehensive prompt template system for evaluating video and image captioning across multiple capability dimensions. Provides system and user prompts for 13 different evaluation tasks with standardized JSON output formats.

Class: Prompts

class Prompts

Central class containing system prompts, user prompt generators, and category definitions for all evaluation tasks.

Task Categories

Video Analysis Tasks

event - Event description evaluation
action - Action mention and recognition
camera_movement - Camera motion classification (7 categories)
dynamic_object_number - Object counting in videos

Image Analysis Tasks

object_category - Object mention and identification
object_number - Object counting in images
object_color - Color attribute accuracy
spatial_relation - Spatial relationship description
scene - Scene identification
camera_angle - Camera angle classification (4 categories)
OCR - Text recognition evaluation
style - Image style classification (9 categories)
character_identification - Person/character naming

System Prompts

event_system_prompt

"You are a video analysis expert specializing in evaluating the accuracy of video captions, particularly the descriptions of the events in a video..."

action_system_prompt

"You are a video analysis expert specializing in evaluating the accuracy of video captions, particularly the descriptions of actions in a video..."

object_category_system_prompt

"You are an image analysis expert specializing in evaluating the accuracy of image captions, particularly the descriptions of objects in an image..."

Additional Prompts

Similar specialized prompts for: object_color, object_number, dynamic_object_number, spatial_relation, scene, camera_angle, camera_movement, OCR, style, character_identification

Category Definitions

camera_angle_categories

level angle - Horizontal shooting (flat shot)
high angle - Shooting from above (overhead shot)
low angle - Shooting from below (upward shot)
dutch angle - Tilted horizon along central axis

camera_movement_categories

left - Pan/track left
right - Pan/track right
up - Tilt/boom up
down - Tilt/boom down
in - Dolly/zoom in
out - Dolly/zoom out
fixed - Static camera

style_categories

realistic - Lifelike detail and accuracy
animated - 2D/3D CGI, cartoon, anime
special effect - Illusions through practical/digital techniques
old-fashioned - Vintage or classical aesthetics
pixel art - Blocky, low-res retro digital art
sketch art - Line work and spontaneity
abstract art - Shapes, colors, emotions over realism
impressionism art - Fleeting light with visible brushstrokes
cubism art - Fragmented geometric planes, multiple perspectives

Key Methods

get_prompts_by_task

def get_prompts_by_task(self, task, caption, anno)

Routes to appropriate prompt generator based on task type.

Parameters:

task - Task identifier (e.g., "event", "action", "camera_angle")
caption - Model-generated caption to evaluate
anno - Ground truth annotation (format varies by task)

Returns: Tuple of (system_prompt, user_prompt)

Raises: ValueError for unsupported task types

Task-Specific Prompt Generators

All generators follow similar pattern:

def get_{task}_prompts(self, caption, anno)

get_event_prompts

Evaluates if event is described in caption.

Scoring:

0 - Totally irrelevant to event
1 - Mentions event correctly
-1 - Mentions relative event with wrong description

Output: JSON with event, score, reason

get_action_prompts

Evaluates action mention in caption.

Scoring:

0 - No actions mentioned
1 - Provided action mentioned
-1 - Provided action not mentioned (but others are)

get_object_category_prompts

Evaluates object mention in caption.

Scoring:

0 - No objects mentioned
1 - Provided object mentioned
-1 - Provided object not mentioned (but others are)

get_object_number_prompts

Evaluates object counting accuracy.

Scoring:

0 - No specific number mentioned or object not mentioned
1 - Correct count
-1 - Wrong count

Input Format: {object: number} dictionary

get_dynamic_object_number_prompts

Similar to object_number but for videos with multiple objects.

Returns: List of user prompts (one per object)

get_object_color_prompts

Evaluates color attribute accuracy.

Scoring:

0 - Object not mentioned OR no color specified
1 - Correct color
-1 - Wrong color

Input Format: {object: color} dictionary

get_spatial_relation_prompts

Evaluates spatial relationship descriptions.

Scoring:

0 - No spatial relationships or objects mentioned
1 - Correct spatial relationship
-1 - Wrong spatial relationship

get_scene_prompts

Evaluates scene identification.

Scoring:

0 - No scene information
1 - Provided scene mentioned
-1 - Different scene mentioned

get_camera_angle_prompts

Classification task for camera angles.

Output: JSON with pred (list of categories) or 'N/A', plus reason

get_camera_movement_prompts

Classification task for camera movements.

Output: JSON with pred (list of categories) or 'N/A', plus reason

Important: Should not infer movement from content, only explicit descriptions

get_OCR_prompts

Evaluates text recognition.

Scoring:

0 - No OCR text description
1 - Text recognized correctly
-1 - Wrong recognition

get_style_prompts

Classification task for image style.

Output: JSON with pred (list of style categories) or 'N/A', plus reason

Focus: Artistic style part of caption

get_character_identification_prompts

Evaluates person/character naming.

Scoring:

0 - No names mentioned
1 - Provided name mentioned correctly
-1 - Wrong name mentioned

Output Format

All tasks use JSON output with variations:

Scoring Tasks:

{"<task>": "<annotation>", "score": <0|1|-1>, "reason": "<explanation>"}

Classification Tasks:

{"pred": [<category_list>] or "N/A", "reason": "<explanation>"}

General Rules:

No markdown syntax
No additional text or explanations
Only the JSON output

Usage Pattern

Instantiate Prompts class
Call get_prompts_by_task with task type, caption, and annotation
Receive system and user prompts
Send to LLM judge
Parse JSON response for score or classification

Dependencies

None - Pure Python class with string templates and category lists

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment