Principle:LaurentMazare Tch rs Seq2Seq Dataset Loading

Knowledge Sources	LaurentMazare_Tch_rs Sequence to Sequence Learning with Neural Networks
Domains	Natural Language Processing, Data Engineering
Last Updated	2026-02-08 00:00 GMT

Overview

Sequence-to-sequence dataset loading prepares paired input-output text sequences for training by performing tokenization, vocabulary construction, length filtering, and optional input reversal.

Description

Sequence-to-sequence (Seq2Seq) models require specially prepared training data where each example consists of a pair of sequences: a source sequence (input) and a target sequence (output). The dataset loading pipeline transforms raw text data through several stages:

• Tokenization: Raw text is split into discrete tokens. For character-level models, each character becomes a token. For word-level models, text is split on whitespace and punctuation. Tokenization must be consistent between training and inference and may involve Unicode normalization, lowercasing, or other text cleaning operations.

• Vocabulary building: A mapping from tokens to integer indices is constructed from the training corpus. The vocabulary records the frequency of each token, enabling vocabulary pruning where infrequent tokens are replaced with a special unknown token. The vocabulary must include special tokens:
  • SOS (Start of Sequence): Signals the beginning of the target sequence during decoding.
  • EOS (End of Sequence): Signals the end of a sequence, telling the decoder when to stop generating.

• Length filtering: Sequences exceeding a maximum length are discarded or truncated. This serves both practical purposes (memory constraints, computational efficiency) and modeling purposes (extremely long sequences are rare and can destabilize training). Filtering is typically applied based on the length of both source and target sequences.

• Input reversal: Optionally, the source sequence is reversed before feeding it to the encoder. This technique, introduced in early Seq2Seq work, places the beginning of the source sequence closer to the beginning of the target sequence in terms of the number of recurrent steps, improving gradient flow and alignment for recurrent encoders.

Usage

Seq2Seq dataset loading is applied in machine translation, text summarization, dialogue generation, question answering, and any task that transforms one sequence into another. The quality of data preparation directly impacts model performance.

Theoretical Basis

Dataset Structure:

A Seq2Seq dataset $D$ consists of paired sequences:

$D=\{(x^{(i)},y^{(i)}){\}}_{i=1}^N$

where $x^{(i)}=(x_1^{(i)},x_2^{(i)},\dots ,x_{T_x}^{(i)})$ is the source sequence and $y^{(i)}=(y_1^{(i)},y_2^{(i)},\dots ,y_{T_y}^{(i)})$ is the target sequence.

Tokenization and Indexing:

A vocabulary function $V:\text{token}\to {\mathbb{\mathbb{Z}}}^{+}$ maps each token to a unique integer:

${\overline{x}}_t^{(i)}=V(x_t^{(i)})$

The indexed target sequence is augmented with special tokens:

${\overline{y}}^{(i)}=(\text{SOS},V(y_1^{(i)}),V(y_2^{(i)}),\dots ,V(y_{T_y}^{(i)}),\text{EOS})$

Length Filtering:

Given maximum lengths $L_{max}^{src}$ and $L_{max}^{tgt}$:

$D_{filtered}=\{(x,y)\in D:|x|\le L_{max}^{src}\text{ and }|y|\le L_{max}^{tgt}\}$

Input Reversal:

The reversed source sequence is:

${\overline{x}}_{rev}^{(i)}=(x_{T_x}^{(i)},x_{T_x-1}^{(i)},\dots ,x_1^{(i)})$

Motivation: For an RNN encoder-decoder, the distance between source token $x_1$ (which typically aligns with target token $y_1$) and the decoder's first step is $T_x+1$ without reversal but only 2 with reversal. This reduces the effective path length for gradient propagation:

$\text{Path length without reversal: }{T}_x+T_y$

$\text{Path length with reversal: }\max (T_x,T_y)+\min (T_x,T_y)\text{ (but first tokens are close)}$

Batching:

Sequences within a batch are padded to the maximum length in the batch:

${\overline{x}}_{padded}^{(i)}=({\overline{x}}_1^{(i)},\dots ,{\overline{x}}_{T_x}^{(i)},\text{PAD},\dots ,\text{PAD})$

A padding mask indicates which positions are real tokens versus padding, preventing the model from attending to or computing loss on padded positions.

Related Pages

• Implementation:LaurentMazare_Tch_rs_Translation_Dataset