Implementation:Speechbrain Speechbrain Train IWSLT22 SAMU mBART

Knowledge Sources	SpeechBrain
Domains	Speech_Translation, Training
Last Updated	2026-02-09 00:00 GMT

Overview

Concrete tool for speech translation using SAMU-pretrained encoder with mBART/NLLB decoder on IWSLT22 provided by the SpeechBrain library.

Description

This recipe defines the ST class (subclass of sb.core.Brain) for fine-tuning a SAMU-pretrained wav2vec2 model combined with the mBART or NLLB decoder for speech translation without transcriptions. The architecture uses wav2vec2 as the encoder with a dimensionality reduction layer, and mBART as the decoder for generating target language text. Supports DistributedDataParallel training, separate optimizers for wav2vec2 and mBART with independent freezing and warmup schedules, and BLEU score evaluation with Moses detokenization.

Usage

Use this recipe to fine-tune a SAMU-pretrained speech translation model with mBART/NLLB decoder on IWSLT22 data. Requires SAMU-pretrained wav2vec2 checkpoint and mBART/NLLB model weights. Configure with train_samu_mbart_st.yaml or train_samu_nllb_st.yaml.

Code Reference

Source Location

• Repository: SpeechBrain
• File: recipes/IWSLT22_lowresource/AST/transformer/train_with_samu_mbart.py

Signature

class ST(sb.core.Brain):
    def compute_forward(self, batch, stage):
        ...
    def compute_objectives(self, predictions, batch, stage):
        ...
    def init_optimizers(self):
        ...
    def freeze_optimizers(self, optimizers):
        ...

Import

python recipes/IWSLT22_lowresource/AST/transformer/train_with_samu_mbart.py hparams/train_samu_mbart_st.yaml --data_folder /path/to/data

I/O Contract

Inputs

Name	Type	Required	Description
batch	PaddedBatch	Yes	Batch containing sig (waveforms), tokens_bos, and tokens_eos (target translation tokens)
stage	sb.Stage	Yes	TRAIN, VALID, or TEST

Outputs

Name	Type	Description
predictions	tuple	Log-softmax sequence probabilities, wav_lens, and decoded hypotheses
loss	torch.Tensor	Sequence-level NLL loss with mBART custom padding applied

Usage Examples

python train_with_samu_mbart.py hparams/train_samu_mbart_st.yaml --data_folder /path/to/data

Related Pages

• Principle:Speechbrain_Speechbrain_Speech_Translation_Training