Implementation:EvolvingLMMs Lab Lmms eval LongVT Utils

Location: /tmp/kapso_repo_sslb_59s/lmms_eval/tasks/longvt/utils.py

Principle: Task_Utility_Functions

Purpose

Task-specific utilities for LongVT (Long Video Test) benchmark with reasoning support and optional tool calling.

Constants

SYSTEM_PROMPT

Instructs model to provide:

• Reasoning process in <think>...</think> tags
• Final answer in <answer>...</answer> tags

TOOL_PROMPT

Additional prompt for tool-based evaluation: "Think first, call **crop_video** if needed, then answer."

Format required: <think>...</think>, optional <tool_call>...</tool_call>, <answer>...</answer>

Configuration

• Reads cache directory from longvt_reasoning.yaml
• Base cache from HF_HOME environment variable

Key Functions

longvt_doc_to_visual

def longvt_doc_to_visual(doc)

Locates and returns video path from cache directory. Asserts video file exists.

longvt_doc_to_text

def longvt_doc_to_text(doc, lmms_eval_specific_kwargs=None)

Formats question text:

• Optionally appends post_prompt from kwargs
• Returns plain question otherwise

longvt_doc_to_messages

def longvt_doc_to_messages(doc, lmms_eval_specific_kwargs=None)

Creates message-based input format:

• System message with SYSTEM_PROMPT
• User message with video and question
• Returns list of message dicts (system + user)

longvt_tool_doc_to_messages

def longvt_tool_doc_to_messages(doc, lmms_eval_specific_kwargs=None)

Creates tool-enabled message format:

• User message with video, question, and TOOL_PROMPT
• No system message (for tool calling mode)
• Returns list with single user message

extract_characters_regex

def extract_characters_regex(s)

Extracts multiple-choice answer (A-D):

• Strips common answer prefixes
• Returns empty string if too verbose without [ABCD]
• Uses regex to find first [ABCD] character

longvt_process_results

def longvt_process_results(doc, results)

Processes results using reasoning_utils:

• Calls compute_score with data_source="longvt"
• Computes acc_score and format_reward_score
• Averages scores across multiple results
• Returns dict with both metrics

Implementation Details

• Supports structured reasoning with think/answer tags
• Optional tool calling capability (crop_video)
• Uses external reasoning_utils for scoring
• Multiple-choice format (A-D)
• Format scoring rewards proper tag usage