Implementation:EvolvingLMMs Lab Lmms eval OpenHermes Utils

Task utility functions for the OpenHermes benchmark, which evaluates audio-based instruction following using GPT-based scoring.

Location

/tmp/kapso_repo_sslb_59s/lmms_eval/tasks/openhermes/utils.py

Overview

Provides audio document processing and GPT-4 based evaluation for OpenHermes tasks. Models are scored on a 0-5 scale by comparing responses to reference answers, with the final score normalized to 0-100.

Core Functions

Document Processing

doc_to_audio(doc)

Extracts audio context from document

Parameters: doc - Document with "context" key

Returns: List containing audio file/path

doc_to_text(doc, lmms_eval_specific_kwargs)

Constructs prompt from pre/post prompts only

Parameters:

• doc - Document (unused)
• lmms_eval_specific_kwargs - Dict with pre_prompt and post_prompt

Returns: "{pre_prompt}{post_prompt}"

Configuration

Module loads openhermes.yaml at import time, filtering out !function lines for safe YAML parsing.

Environment Variables

• MODEL_VERSION: GPT model name (default: "gpt-4o-2024-11-20")
• API_TYPE: "openai" or "azure" (default: "azure")

OpenAI Configuration:

• OPENAI_API_URL: API endpoint (default: OpenAI completions URL)
• OPENAI_API_KEY: API key
• Headers: Authorization: Bearer {API_KEY}

Azure Configuration:

• AZURE_ENDPOINT: Azure endpoint URL
• AZURE_API_KEY: API key
• Headers: api-key: {API_KEY}

GPT Evaluation

get_eval(max_tokens, content, retries=3)

Calls GPT API with retry logic

Parameters:

• max_tokens (int): Maximum tokens for response
• content (str): Evaluation prompt
• retries (int): Number of retry attempts (default 3)

Process:

1. Constructs user message with content
2. Creates payload with model, temperature (0.7), top_p (0.95)
3. POSTs to API with 60s timeout
4. On success: extracts and returns response content + model name
5. On failure: sleeps 5s and retries

Returns: Tuple of (response_content, model_name) or ("", "") on failure

Retry Behavior

• NUM_SECONDS_TO_SLEEP = 5
• Logs each failed attempt
• Returns empty strings after all retries exhausted

Result Processing

openhermes_process_results(doc, result)

Evaluates model response using GPT

Parameters:

• doc - Document with answer and speech_instruction
• result - Model prediction list

Process:

1. Extracts prediction from result[0]
2. Formats evaluation prompt with question, reference, and model response
3. Calls get_eval with max_tokens=1024

Returns: Dictionary with gpt_eval entry containing:

• eval_answer: GPT evaluation response
• model_name: GPT model used

Aggregation

openhermes_aggregate_results(results)

Computes average normalized score across all results

Parameters: results - List of result dicts with eval_answer

Process:

1. For each result:
   1. Extracts numeric rating (0-5) using regex ([0-5])
   2. Converts to float
   3. On parse error: defaults to 0.0 and logs error
2. Sums all scores
3. Computes mean
4. Multiplies by 20 to normalize to 0-100 scale

Returns: Normalized score (0-100)

Evaluation Prompt

The module defines a detailed evaluation prompt template with:

Sections:

• [Question]: Original instruction
• [Reference Answer]: Ground truth
• [Model Answer]: Model's response

Task: Rate alignment with reference on accuracy and relevance.

Scoring Rubric:

• Score 0: Completely misaligned
• Score 1: Minimal alignment, irrelevant details
• Score 2: Recognizes topic but diverges significantly
• Score 3: General alignment but lacks detail/precision
• Score 4: Mostly accurate and relevant, could be clearer
• Score 5: Highly accurate, detailed, perfect match

Response Format:

Explanation: (Comparison reasoning)
Rating: (int)

Dependencies

• os, re, time, pathlib.Path
• requests - HTTP client for API calls
• yaml - Config file parsing
• loguru.logger as eval_logger

Constants

• retries = 3 - Global retry count
• NUM_SECONDS_TO_SLEEP = 5 - Sleep duration between retries

Error Handling

• API failures are logged with attempt number
• Final failure after all retries is logged as error
• Parse errors in score extraction default to 0.0 with error log

Related

• Task_Utility_Functions - General task utility pattern
• GPT_Evaluation - GPT-based evaluation pattern
• Audio_Task_Utils - Audio processing utilities