Principle:Tencent Ncnn Audio Preprocessing
| Knowledge Sources | |
|---|---|
| Domains | Signal Processing, Speech Recognition |
| Last Updated | 2026-02-09 19:00 GMT |
Overview
The conversion of raw audio waveforms into spectral feature representations, specifically log-Mel spectrograms, through sequential application of windowing, Fast Fourier Transform, and Mel-scale filterbank projection.
Description
Audio preprocessing transforms a one-dimensional time-domain audio signal into a two-dimensional time-frequency representation suitable for consumption by neural networks. This transformation is essential because raw waveform samples contain enormous amounts of redundant information, while spectral features compactly encode the perceptually relevant frequency content at each point in time.
The pipeline proceeds in three stages. First, the audio signal is divided into short overlapping segments using a window function (typically a Hann window). Each segment, or frame, captures a snapshot of the signal's frequency content over a duration of approximately 25 milliseconds, with frames overlapping by about 10 milliseconds to ensure temporal continuity.
Second, each windowed frame is transformed from the time domain to the frequency domain via the Fast Fourier Transform (FFT). The squared magnitude of the FFT output yields the power spectrum, which describes how energy is distributed across frequency bins for that frame.
Third, the power spectrum is projected onto a Mel filterbank, a set of triangular bandpass filters spaced according to the Mel scale. The Mel scale is a perceptual scale that approximates the human ear's frequency resolution, spacing filters more closely at low frequencies and more widely at high frequencies. Taking the logarithm of the filterbank energies yields the log-Mel spectrogram, which compresses the dynamic range and better matches human auditory perception.
Usage
This principle applies as the mandatory first stage in any audio-based inference pipeline:
- Speech recognition: Converting spoken audio into features for transcription models.
- Speaker identification: Extracting voice characteristics for identity verification.
- Audio classification: Identifying sounds, music genres, or environmental audio events.
- Keyword spotting: Detecting wake words or specific spoken commands.
Theoretical Basis
The windowing and short-time Fourier transform:
Given:
signal = raw audio samples at sample_rate (e.g., 16000 Hz)
n_fft = FFT window size (e.g., 400 samples = 25ms at 16kHz)
hop_length = frame shift (e.g., 160 samples = 10ms at 16kHz)
window = hann_window(n_fft)
For each frame i:
frame = signal[i * hop_length : i * hop_length + n_fft]
windowed_frame = frame * window
spectrum = FFT(windowed_frame)
power_spectrum = |spectrum|^2
The Mel filterbank projection and log compression:
// Mel scale conversion
mel(f) = 2595 * log10(1 + f / 700)
mel_inv(m) = 700 * (10^(m / 2595) - 1)
// Create n_mels triangular filters spanning [f_min, f_max]
mel_filters = triangular_filterbank(n_mels, n_fft, sample_rate, f_min, f_max)
// Apply filterbank to each frame's power spectrum
mel_energies = mel_filters @ power_spectrum // shape: (n_mels,)
// Log compression with numerical stability
log_mel = log(max(mel_energies, 1e-10))
The resulting log-Mel spectrogram has shape where is the number of time frames and is the number of Mel frequency bins (commonly 80 or 128).