Implementation:EvolvingLMMs Lab Lmms eval People Speech Utils

Task utility functions for the People Speech benchmark, which evaluates automatic speech recognition across multiple languages and sources using Word Error Rate (WER).

Location

lmms_eval/tasks/people_speech/utils.py

Overview

Provides audio processing, result handling, and WER computation for People Speech tasks. Supports multi-language evaluation (English, Chinese, Yue) with language-specific text normalization and tokenization.

Core Functions

Document Processing

people_speech_doc_to_audio(doc)

Extracts audio from document

Parameters: doc - Document with "audio" key

Returns: List containing audio file/data

people_speech_doc_to_text(doc, lmms_eval_specific_kwargs)

Constructs fixed ASR prompt

Parameters: doc, lmms_eval_specific_kwargs (with prompts)

Returns: "{pre_prompt}Please recognize the speech and only output the recognized content:{post_prompt}"

Result Processing

people_speech_process_result(doc, result)

Packages prediction with metadata for WER computation

Parameters:

• doc - Document with gt, source, task
• result - Model prediction list

Returns: Dictionary with wer entry containing:

• gt: Ground truth transcription
• pred: Model prediction
• source: Data source
• task: Task identifier (contains language code)

Text Normalization

remove_sp(text, language)

Removes special tokens and normalizes spacing

Parameters:

• text - Input text
• language - Language code ("zh", "en", etc.)

Process:

1. Removes tokens matching <|.*|> pattern
2. Collapses consecutive spaces to single space
3. Removes space before punctuation (!,.?;:)
4. Left-strips whitespace
5. For Chinese ("zh"): removes all remaining spaces

Returns: Normalized text string

EvaluationTokenizer Class

Language-aware tokenizer leveraging sacreBLEU tokenizers.

Initialization

EvaluationTokenizer(
    tokenizer_type="13a",
    lowercase=False,
    punctuation_removal=False,
    character_tokenization=False
)

Parameters

• tokenizer_type: Tokenizer variant - "none", "13a", "intl", "zh", "ja-mecab", "char"
• lowercase: Apply lowercasing
• punctuation_removal: Remove punctuation-only tokens
• character_tokenization: Character-level tokenization

Class Constants

• SPACE = chr(32)
• SPACE_ESCAPE = chr(9601)

Methods

remove_punctuation(sent) (classmethod)

Removes tokens that are purely punctuation

Parameters: sent - Space-separated string

Process: Filters tokens where all characters are Unicode category "P" (punctuation)

Returns: String with punctuation-only tokens removed

tokenize(sent)

Applies full tokenization pipeline

Parameters: sent - Input string

Process:

1. Apply sacreBLEU tokenizer
2. Optionally remove punctuation
3. Optionally tokenize to characters (with space escape)
4. Optionally lowercase

Returns: Tokenized string

WER Computation

compute_wer(refs, hyps, language)

Computes Word Error Rate using edit distance.

Parameters:

• refs - List of reference transcriptions
• hyps - List of hypothesis transcriptions
• language - Language code

Process:

1. For each reference-hypothesis pair:
   1. Apply language-specific normalization:
      • yue: Convert to simplified Chinese via zhconv.convert(text, "zh-cn")
      • en: Apply english_normalizer (Whisper English normalizer)
      • zh: Apply chinese_normalizer (TextNorm with custom config)
      • Other: Apply basic_normalizer (Whisper basic normalizer)
   2. Tokenize using EvaluationTokenizer:
      • Type: "none"
      • Lowercase: True
      • Punctuation removal: True
      • Character tokenization: False
   3. For Chinese/Yue languages:
      • Join tokenized items
      • Character-level split
   4. For first pair (i==0): print debug info
   5. Compute edit distance between token sequences
   6. Accumulate distance and reference length
2. Return: total_distance / total_ref_length

Returns: WER as decimal (0-1)

people_speech_wer(results, args)

Aggregates WER across all results with language extraction.

Parameters:

• results - List of result dicts with gt, pred, task
• args - Arguments (unused)

Process:

1. For each result:
   • Extract language code from task field: task[4:] (assumes format "asr_en", "asr_zh", etc.)
   • Extract ground truth and prediction
   • Apply remove_sp normalization
   • Append to refs/hyps lists
2. Compute WER via compute_wer(refs, hyps, lan)
3. Multiply by 100 for percentage

Returns: WER percentage (0-100)

Global Normalizers

Initialized at module level:

• english_normalizer: EnglishTextNormalizer()
• chinese_normalizer: TextNorm(...) with specific configuration
• basic_normalizer: BasicTextNormalizer()

Chinese Normalizer Configuration:

TextNorm(
    to_banjiao=False,
    to_upper=False,
    to_lower=False,
    remove_fillers=False,
    remove_erhua=False,
    check_chars=False,
    remove_space=False,
    cc_mode=""
)

Dependencies

• os, re, unicodedata
• editdistance as ed - Edit distance computation
• zhconv - Chinese variant conversion (simplified/traditional)
• lmms_eval.tasks.librispeech.cn_tn.TextNorm - Chinese text normalization
• lmms_eval.tasks.librispeech.whisper_normalizer.basic.BasicTextNormalizer
• lmms_eval.tasks.librispeech.whisper_normalizer.english.EnglishTextNormalizer
• sacrebleu.tokenizers.* - Multiple tokenizer implementations

Constants

• PUNCS = "!,.?;:" - Punctuation characters for normalization
• dir_name - Absolute path to module directory

Design Differences from Open ASR

While similar to Open_ASR_Utils, this implementation:

• Extracts language from task field (format: "asr_{lang}")
• Includes source and task in result dictionaries
• Assumes specific task naming convention for language extraction
• Single WER computation across all results (no per-source breakdown)

Related

• Task_Utility_Functions - General task utility pattern
• Open_ASR_Utils - Similar ASR utilities
• LibriSpeech_Utils - Speech recognition utilities
• WER_Metric - Word Error Rate metric details
• Multi_Language_Evaluation - Multi-language evaluation patterns