Principle:Huggingface Datasets Sequence Feature Handling
| Knowledge Sources | |
|---|---|
| Domains | Data_Engineering, NLP |
| Last Updated | 2026-02-14 18:00 GMT |
Overview
Representing variable or fixed-length sequences of feature types enables datasets to store list-valued columns such as token IDs, embeddings, or multi-label annotations.
Description
Sequence feature handling allows dataset columns to contain lists (sequences) of values of a given feature type. A sequence can be variable-length (the default, with length=-1) or fixed-length. The feature wraps any other feature type (Value, ClassLabel, nested dicts, etc.) to create a list column. A key compatibility behavior is that when a Sequence wraps a dictionary of features, it automatically converts to a dictionary of lists (rather than a list of dictionaries), maintaining compatibility with the TensorFlow Datasets library. For users who do not want this transposition behavior, the List or LargeList types can be used directly.
Usage
Use sequence features when a column should contain variable-length lists of items, such as token ID sequences, embedding vectors, bounding box coordinates, multi-label classifications, or any repeated sub-structure. Choose fixed length when all sequences in the column have the same size (e.g., fixed-dimension embeddings).
Theoretical Basis
Sequences are backed by Arrow's list type, which stores variable-length arrays of a child type efficiently using offset arrays. The Sequence class acts as a factory: when the child feature is a dict of sub-features, it produces a dict of List features (transposition for TFDS compatibility); otherwise, it creates a standard List. This design allows a single Sequence declaration to handle both flat sequences (e.g., Sequence(Value("int32"))) and structured sequences (e.g., Sequence({"start": Value("int32"), "end": Value("int32")})) with different underlying representations.