Implementation:Speechbrain Speechbrain Train SLURP Direct Wav2Vec

Knowledge Sources	SpeechBrain
Domains	SLU, Training
Last Updated	2026-02-09 00:00 GMT

Overview

Concrete tool for direct (speech to semantics) SLU training using wav2vec2/HuBERT on the SLURP dataset provided by the SpeechBrain library.

Description

This recipe defines the SLU class (subclass of sb.Brain) for direct spoken language understanding on the SLURP dataset using wav2vec2 or HuBERT as the speech encoder. Input waveforms are encoded directly with wav2vec2, then fed into a seq2seq decoder to produce semantic token outputs. Unlike the ASR-transfer variant, this recipe fine-tunes the wav2vec2 model directly without a separate SLU encoder layer. Supports waveform augmentation with label replication, beam search decoding, and SLURP-specific evaluation metrics.

Usage

Use this recipe to train a direct SLU model on the SLURP dataset using wav2vec2/HuBERT features. Requires the SLURP data folder and a pre-trained wav2vec2 model from HuggingFace. Configure with hparams/train_with_wav2vec2.yaml.

Code Reference

Source Location

• Repository: SpeechBrain
• File: recipes/SLURP/direct/train_with_wav2vec2.py

Signature

class SLU(sb.Brain):
    def compute_forward(self, batch, stage):
        ...
    def compute_objectives(self, predictions, batch, stage):
        ...

Import

python recipes/SLURP/direct/train_with_wav2vec2.py hparams/train_with_wav2vec2.yaml --data_folder /path/to/SLURP

I/O Contract

Inputs

Name	Type	Required	Description
batch	PaddedBatch	Yes	Batch containing sig (waveforms), tokens_bos, and tokens_eos (semantic tokens)
stage	sb.Stage	Yes	TRAIN, VALID, or TEST

Outputs

Name	Type	Description
predictions	tuple	Log-softmax sequence probabilities, wav_lens, and decoded hypotheses
loss	torch.Tensor	Sequence-level NLL loss on semantic tokens

Usage Examples

python train_with_wav2vec2.py hparams/train_with_wav2vec2.yaml --data_folder /path/to/SLURP

Related Pages

• Principle:Speechbrain_Speechbrain_Spoken_Language_Understanding