Implementation:EvolvingLMMs Lab Lmms eval OVOBench Utils

Task utility functions for the OVOBench (Online Video Object benchmark), which evaluates video understanding with forward, backward, and realtime task types.

Location

/tmp/kapso_repo_sslb_59s/lmms_eval/tasks/ovobench/utils.py

Overview

Provides video processing, prompt building, and accuracy computation for OVOBench tasks. Supports three task categories with different temporal reasoning modes: forward (streaming), backward (full video), and realtime (live scenarios).

Task Categorization

Task Types

Forward Tasks (streaming/progressive):

• REC - Repetition Counting
• SSR - Step Sequence Recognition
• CRR - Contextual Reasoning Recognition

Backward Tasks (full video retrospective):

• EPM - Event Prediction Modeling
• ASI - Action Sequence Identification
• HLD - High-Level Description

Realtime Tasks (live/immediate):

• STU - Situation Understanding
• OJR - Object Judgment Recognition
• ATR - Action Temporal Recognition
• ACR - Activity Classification Recognition
• OCR - Object Classification Recognition
• FPD - Future Prediction Decision

Helper Functions

is_forward_task(doc)

Determines if task is forward type

Parameters: doc - Document with "task" key

Returns: True if task NOT in backward/realtime list

get_task_type(task_name)

Maps task name to category

Parameters: task_name - Task abbreviation

Returns: "backward", "realtime", or "forward"

Raises: ValueError for unknown task names

Prompt Building

build_prompt(doc, index)

Constructs task-specific prompt

Parameters:

• doc - Document with task, question, options, activity, test_info
• index - Round index (for SSR tasks)

Backward/Realtime Tasks:

• Extracts question and options
• Formats options as "A. option1; B. option2; ..."
• Uses BR_PROMPT_TEMPLATE

Forward Tasks:

• REC: "How many times did they {activity}?" with REC_PROMPT_TEMPLATE
• SSR: Uses step from test_info[index]["step"] with SSR_PROMPT_TEMPLATE
• CRR: Uses question with CRR_PROMPT_TEMPLATE

Returns: Formatted prompt string

Document Processing

Backward/Realtime Tasks

ovo_back_real_doc_to_text(doc, lmms_eval_specific_kwargs=None)

Generates flat text prompt

Parameters: doc, lmms_eval_specific_kwargs (unused)

Process: Calls build_prompt(doc, index=None)

Returns: Prompt string

Forward Tasks (Multi-Round)

ovo_forward_doc_to_text(doc, lmms_eval_specific_kwargs=None, previous_output=None, round_idx=None, previous_round_info=None)

Assembles prompt/visual payloads for progressive rounds

Parameters:

• doc - Document
• lmms_eval_specific_kwargs - Kwargs for data directory
• previous_output - Output from previous round
• round_idx - Current round index (None for initial round)
• previous_round_info - Info from previous round

Initial Round (round_idx=None):

• Builds prompt with index=0
• Returns prompt string

Subsequent Rounds:

• Builds prompt for round_idx
• Retrieves visuals via ovo_doc_to_visual
• Returns tuple: (visuals, prompt, terminal_sign, previous_output, None)

Terminal Round:

• When round_idx == len(doc["test_info"])
• Returns: (None, None, True, previous_output, None)

Returns: Varies by round stage

Visual Processing

ovo_doc_to_visual(doc, lmms_eval_specific_kwargs)

Returns video chunk path for document/round

Parameters:

• doc - Document with "id"
• lmms_eval_specific_kwargs - Must contain data_dir and optionally round_idx

Path Construction:

• With round_idx: {data_dir}/{id}_{round_idx}.mp4
• Forward task without round_idx: {data_dir}/{id}_0.mp4
• Backward/realtime: {data_dir}/{id}.mp4

Returns: List containing video chunk path

Assertions: data_dir not None, video path exists

Result Processing

Backward/Realtime

ovo_back_real_process_results(doc, results)

Normalizes output into structured record

Parameters:

• doc - Document
• results - Model predictions (handles nested lists)

Process:

1. Extracts response, handling both flat and nested list formats
2. Strips whitespace
3. Converts ground truth index to letter (A, B, C, ...)

Returns: Dictionary with back_real_acc entry containing:

• id, task, question, response, ground_truth

Forward

ovo_forward_process_results(doc, results)

Maps multi-round responses to document structure

Parameters:

• doc - Document with test_info list
• results - Model predictions (handles nested lists)

Process:

1. Extracts results list
2. For each round, assigns test_info[i]["response"] = results[i]

Returns: Dictionary with forward_acc entry containing modified doc

Aggregation and Scoring

Backward/Realtime Accuracy

ovo_back_real_acc(results, args)

Scores backward/realtime outputs and saves results

Parameters:

• results - List of result dictionaries
• args - Arguments for file generation

Process:

1. Calls calculate_score_backward_realtime(results)
2. Determines task type from first score key
3. Saves detailed results to JSON
4. Computes score summary:
   1. Per-task accuracy: 100 * sum(scores) / len(scores)
   2. Average across tasks
5. Saves summary JSON and Excel

Returns: Average accuracy percentage (0-100)

Output Files:

• {task_name}_acc_results.json - Detailed results
• {task_name}_acc_scores.json - Score summary with task-level and average

Forward Accuracy

ovo_forward_acc(results, args)

Scores forward outputs and saves results

Parameters: results, args

Process:

1. Calls calculate_score_forward(results)
2. Saves to forward_acc_results.json
3. Computes and saves scores to forward_acc_scores.json

Returns: Average accuracy percentage (0-100)

Output Files:

• forward_acc_results.json
• forward_acc_scores.json

Multi-Round Notes

ovo_doc_to_messages(doc, lmms_eval_specific_kwargs)

Placeholder for multi-round chat generation (not yet implemented)

Note: Multi-round generation for chat models not implemented

Dependencies

• json, os
• lmms_eval.tasks._task_utils.file_utils.generate_submission_file
• lmms_eval.tasks.ovobench.constant:
  • BR_PROMPT_TEMPLATE, CRR_PROMPT_TEMPLATE, REC_PROMPT_TEMPLATE, SSR_PROMPT_TEMPLATE
• lmms_eval.tasks.ovobench.score_utils.score:
  • calculate_score_backward_realtime, calculate_score_forward

Related

• Task_Utility_Functions - General task utility pattern
• OVOBench_Constants - Prompt templates
• OVOBench_Scoring - Score calculation implementations
• Multi_Round_Generation - Multi-round task pattern