Principle:Openai Whisper Decoding Configuration
Overview
Decoding Configuration defines the strategy for autoregressive sequence generation in speech recognition. When a Whisper model produces text from audio, the decoder generates tokens one at a time, each conditioned on all previously generated tokens. The decoding configuration controls how this generation proceeds: which tokens are considered, how they are ranked, and when generation stops.
These are standard hyperparameters in sequence-to-sequence models, applied here to the specific domain of automatic speech recognition (ASR).
Decoding Strategies
The core decoding choice is between deterministic and stochastic generation:
Greedy Decoding
When temperature = 0.0, the decoder selects the single highest-probability token at each step. This is the fastest strategy and produces deterministic output. Greedy decoding is the default in Whisper because speech recognition typically has a single correct answer, unlike open-ended text generation.
Sampling
When temperature > 0.0, the logits are divided by the temperature before applying softmax, producing a flattened probability distribution. A token is then sampled from this distribution rather than deterministically chosen. Higher temperatures produce more diverse but less reliable outputs.
The best_of parameter controls n-best sampling: multiple sequences are generated independently and the one with the highest average log probability is selected. This improves quality at the cost of additional computation.
Beam Search
Beam search maintains multiple candidate sequences (beams) simultaneously, expanding each by one token per step and pruning to keep only the top-scoring candidates. The beam_size parameter controls how many beams are maintained. Beam search with patience allows beams to terminate at different times, waiting for potentially better completions.
Note: Beam search and n-best sampling are mutually exclusive strategies. You cannot set both beam_size and best_of simultaneously.
Task Selection
Whisper supports two tasks:
- "transcribe" — Transcribes speech in the source language (X to X). This is the default.
- "translate" — Translates speech from any language into English (X to English).
The task is encoded as a special token in the decoder's initial prompt sequence, directing the model's behavior without any architectural changes.
Language Specification
The language parameter specifies the spoken language using a language code (e.g., "en", "ja", "fr"). When set to None, the model performs automatic language detection by examining the probability distribution over language tokens after processing the audio through the encoder. Specifying the language explicitly skips this detection step and can improve accuracy when the language is known in advance.
Token Suppression
Token suppression prevents the decoder from generating unwanted tokens:
- suppress_tokens — A set of token IDs that are assigned negative infinity logits, making them impossible to select. The default value "-1" maps to a predefined list of non-speech symbols (e.g., music notes, special characters) that should not appear in transcribed text.
- suppress_blank — Prevents the model from generating blank or end-of-text tokens at the very start of decoding, forcing it to produce at least some content.
Timestamp Control
Whisper uses special timestamp tokens interleaved with text tokens to provide time alignment. The without_timestamps parameter disables these tokens entirely when timing information is not needed. The max_initial_timestamp parameter constrains the first timestamp to be within a certain number of seconds from the start of the segment, preventing the decoder from skipping large portions of audio.
Precision Settings
The fp16 parameter controls whether computation uses half-precision (16-bit) floating point. Half-precision is faster and uses less memory on CUDA GPUs, but is not supported on CPU. This is a hardware-level optimization that does not change the decoding logic.
Prompt and Prefix
- prompt — Text from a previous decoding segment or user-provided context, placed before the start-of-transcript token. This conditions the model on prior context for consistency across segments.
- prefix — Text placed after the start-of-transcript token, forcing the decoder to begin with specific tokens. Useful for guiding the initial output.
Length Control
- sample_len — Maximum number of tokens to generate before stopping.
- length_penalty — Applied during beam search to adjust scores based on sequence length, preventing bias toward shorter or longer outputs.
References
- Radford, A., Kim, J.W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2022). Robust Speech Recognition via Large-Scale Weak Supervision. arXiv:2209.11302