Implementation:Speechbrain Speechbrain Train IWSLT22 W2V mBART

Knowledge Sources	SpeechBrain
Domains	Speech_Translation, Training
Last Updated	2026-02-09 00:00 GMT

Overview

Concrete tool for speech translation using wav2vec2 encoder with mBART/NLLB decoder on IWSLT22 provided by the SpeechBrain library.

Description

This recipe defines the ST class (subclass of sb.core.Brain) for fine-tuning a wav2vec2 model combined with the mBART or NLLB decoder for speech translation without transcriptions. Unlike the SAMU variant, this uses standard wav2vec2 without SAMU pretraining. The architecture encodes speech with wav2vec2, applies dimensionality reduction, and decodes with mBART. Supports DistributedDataParallel training, separate optimizers for wav2vec2 and mBART with independent warmup and freezing schedules, and BLEU score evaluation with Moses detokenization.

Usage

Use this recipe to train a speech translation model combining wav2vec2 and mBART/NLLB on IWSLT22 data. Requires pre-trained wav2vec2 and mBART/NLLB model weights. Configure with train_w2v2_mbart_st.yaml or train_w2v2_nllb_st.yaml.

Code Reference

Source Location

• Repository: SpeechBrain
• File: recipes/IWSLT22_lowresource/AST/transformer/train_with_w2v_mbart.py

Signature

class ST(sb.core.Brain):
    def compute_forward(self, batch, stage):
        ...
    def compute_objectives(self, predictions, batch, stage):
        ...
    def init_optimizers(self):
        ...
    def freeze_optimizers(self, optimizers):
        ...

Import

python recipes/IWSLT22_lowresource/AST/transformer/train_with_w2v_mbart.py hparams/train_w2v2_mbart_st.yaml --data_folder /path/to/data

I/O Contract

Inputs

Name	Type	Required	Description
batch	PaddedBatch	Yes	Batch containing sig (waveforms), tokens_bos, and tokens_eos (target translation tokens)
stage	sb.Stage	Yes	TRAIN, VALID, or TEST

Outputs

Name	Type	Description
predictions	tuple	Log-softmax sequence probabilities, wav_lens, and decoded hypotheses
loss	torch.Tensor	Sequence-level NLL loss with mBART custom padding applied

Usage Examples

python train_with_w2v_mbart.py hparams/train_w2v2_mbart_st.yaml --data_folder /path/to/data

Related Pages

• Principle:Speechbrain_Speechbrain_Speech_Translation_Training