Principle:Openai Whisper Word Boundary Detection

Overview

Word Boundary Detection is the process of segmenting a sequence of subword tokens back into word-level groups. Modern speech recognition systems like Whisper use BPE (Byte Pair Encoding) or similar subword tokenization schemes that split text into subword units which do not necessarily correspond to word boundaries. To produce word-level timestamps, these subword tokens must be re-grouped into words using language-appropriate strategies.

Domain

• Natural Language Processing
• Tokenization

The Subword Tokenization Problem

BPE and similar subword tokenizers split text into variable-length subword units based on frequency statistics. For example, the word "unfortunately" might be tokenized as:

"unfortunately" -> ["un", "fortunate", "ly"]

Each subword token has its own token ID and, after alignment, its own timestamp. However, users expect word-level timestamps, not subword-level ones. Therefore, subword tokens must be re-grouped into words.

Language-Dependent Strategies

Word boundary detection is inherently language-dependent because languages differ in how they delimit words:

Space-Delimited Languages

Languages such as English, French, German, Spanish, and most European languages use whitespace to separate words. In these languages, BPE tokens that represent the beginning of a word typically start with a space character (or a special space token). The word boundary detection strategy is:

1. Decode each token to its text representation.
2. Merge consecutive tokens that do not start with a space character into the same word group.
3. Tokens that start with a space begin a new word group.

For example:

Tokens: [" Hello", " world", ",", " how"]
Words:  [" Hello", " world,", " how"]

Note that punctuation without a leading space (like the comma) merges with the preceding word.

CJK and Non-Space Languages

Languages such as Chinese, Japanese, Thai, Lao, and Burmese do not use spaces between words. For these languages, a unicode-based splitting strategy is used:

1. Decode each token to its unicode representation.
2. Split at each valid unicode code point boundary.
3. Each resulting character (or small cluster) becomes its own "word."

This produces character-level rather than word-level groupings, which is appropriate for these languages since:

• Chinese characters are typically single-morpheme units.
• Japanese kanji/kana characters often function as individual meaningful units.
• Thai and similar scripts require specialized word segmentation that is beyond the scope of the tokenizer.

Detection Mechanism

The detection mechanism uses a two-step process:

1. Unicode splitting: First, all tokens are decoded and split at unicode character boundaries. This handles all languages uniformly at the character level.
2. Space-based merging (for applicable languages): Characters/subwords that do not begin with a space are merged with their preceding element, reconstructing space-delimited words.

This approach ensures:

• Robustness: Works correctly even when tokens span multiple characters or contain mixed scripts.
• Language coverage: Handles both space-delimited and non-space languages.
• Consistency: The same tokenizer and splitting logic is used for all languages, with only the merge behavior varying.

Significance for Timestamps

Accurate word boundary detection is essential for:

• Word-level timestamps: Each word's start and end time is determined by the timestamps of its constituent tokens.
• Subtitle generation: Subtitles are displayed at the word level, not the subword level.
• Punctuation handling: Proper word grouping feeds into downstream punctuation merging.

Implementation

Implementation:Openai_Whisper_Split_To_Word_Tokens