Implementation:EvolvingLMMs Lab Lmms eval Charades STA Eval TVG

File: lmms_eval/tasks/charades_sta/eval_tvg.py (147 lines)

Principle: Task Utility Functions

Overview

Evaluation script for Temporal Video Grounding (TVG) on the Charades-STA dataset. Extracts temporal timestamps from natural language responses, computes IoU (Intersection over Union) against ground truth, and calculates accuracy at multiple IoU thresholds.

Key Functions

read_json

def read_json(path)

Loads JSON data from file.

Parameters:

• path - File path to JSON file

Returns: Parsed JSON data

write_json

def write_json(path, data)

Saves data to JSON file with indentation.

Parameters:

• path - Output file path
• data - Data to serialize

Side Effects: Prints save confirmation message

extract_time

def extract_time(paragraph)

Extracts temporal timestamps from natural language text using multiple regex patterns.

Parameters:

• paragraph - Text containing temporal information

Returns: List of [start, end] timestamp pairs (may be empty)

Preprocessing:

• Converts to lowercase
• Removes example prompt: "A specific example is : 20.8 - 30.0 seconds"
• Replaces "to" with "-"

Keyword Filtering: Filters sentences containing temporal keywords:

• starts, ends, happens in, start time, end time, start, end, happen

Pattern Matching (in priority order):

1. Main Pattern: (\d+(?:\.\d+)?)\s*[–-]\s*(\d+(?:\.\d+)?)

• • Matches: "20.8 - 30.0", "5-10", "3.5–7.2"
  • Returns all matches as [start, end] pairs

1. Time Number Pattern: \b(\d+(?:\.\d+)?)\b

• • Extracts individual numbers from sentences with keywords
  • Pairs consecutive numbers: times[0] with times[1], times[2] with times[3], etc.
  • Example: "Starting time: 0.8 seconds Ending time: 1.1 seconds"

1. Time Format Pattern: \b(\d{1,2}:\d{2}:\d{2})\b

• • Matches HH:MM:SS or MM:SS format
  • Converts to seconds: h*3600 + m*60 + s or m*60 + s
  • Pairs consecutive timestamps

1. Fallback Pattern: (\d+(?:\.\d+)?)\s*s?\s*[–-]\s*(\d+(?:\.\d+)?)\s*s?

• • Matches various formats with optional 's' suffix
  • Handles both en dash (–) and regular dash (-)

Post-processing:

• Ensures start < end (swaps if necessary)
• Takes only first timestamp pair if multiple found
• Returns empty list if no valid timestamps extracted

iou

def iou(A, B)

Calculates Intersection over Union between two temporal segments.

Parameters:

• A - First segment [start, end]
• B - Second segment [start, end]

Returns: IoU value (0.0 to 1.0)

Formula:

IoU = max(0, overlap) / union
overlap = min(A[1], B[1]) - max(A[0], B[0])
union = max(A[1], B[1]) - min(A[0], B[0])

Main Execution

When run as script with __name__ == "__main__":

Command-line Arguments

parser.add_argument("-f", default="your_result.json")

• -f - Path to results JSON file

Results Format

Expected dictionary structure:

{
  "vid>>>caption>>>gt": "model_response",
  ...
}

Where:

• vid - Video identifier
• caption - Query/caption text
• gt - Ground truth [start, end] as string (parseable with eval())
• model_response - Natural language response containing predicted timestamps

Evaluation Process

1. Loads results from JSON file
2. Iterates through each entry
3. Splits key into vid, caption, gt components
4. Parses ground truth using eval()
5. Extracts predicted timestamps using extract_time()
6. Fallback: If extraction fails or returns != 1 timestamp:
   • Uses [gt[1] + 10, gt[1] + 20] as dummy prediction
   • Prints error message with prediction and extracted timestamps
7. Calculates IoU between ground truth and prediction
8. Collects all IoU values

Metrics

Calculates accuracy at three IoU thresholds:

• IoU@0.3 - Percentage of predictions with IoU ≥ 0.3
• IoU@0.5 - Percentage of predictions with IoU ≥ 0.5
• IoU@0.7 - Percentage of predictions with IoU ≥ 0.7
• mIoU - Mean IoU across all predictions

Output Format:

IOU 0.3: {percentage}
IOU 0.5: {percentage}
IOU 0.7: {percentage}
mIOU {mean_iou}

Usage Example

python eval_tvg.py -f results.json

Example Results Entry:

Key: "video_001>>>person walking>>>\"[5.2, 12.8]\""
Value: "The event starts at 5.5 seconds and ends at 12.5 seconds."

Extraction: [5.5, 12.5] Ground Truth: [5.2, 12.8] IoU: ~0.87

Dependencies

• argparse - Command-line argument parsing
• json - JSON file I/O
• re - Regular expression matching for timestamp extraction