Implementation:Huggingface Datasets TenBin
| Knowledge Sources | |
|---|---|
| Domains | Binary_Encoding, Tensor_Serialization |
| Last Updated | 2026-02-14 18:00 GMT |
Overview
Binary tensor encoding/decoding library for the .ten format used by WebDataset.
Description
TenBin (short for "Tensor Binary") is an internal utility module that implements efficient binary encoding and decoding of NumPy arrays. It originates from the WebDataset library (NVIDIA, BSD-style license) and is bundled within HuggingFace Datasets to support the WebDataset packaged module.
The binary format is 8-byte aligned and structured as a series of chunks, each consisting of:
- A magic number (int64,
"~TenBin~") - A length field in bytes (int64)
- Data bytes padded to a multiple of 64 bytes
Each array is serialized as a header chunk followed by a data chunk. The header encodes the dtype (as a 2-character short code), an optional 8-character info string, the number of dimensions, and the shape. Supported dtypes include float16, float32, float64, int8, int16, int32, int64, uint8, uint16, uint32, and uint64.
The module provides functions at multiple abstraction levels: low-level chunk encoding/decoding, list-based array encoding/decoding, buffer-based operations, stream-based I/O, and file-based save/load convenience functions.
Usage
TenBin is not imported directly by users. It is used internally by the WebDataset builder to deserialize tensor data stored in .ten files within WebDataset tar archives.
Code Reference
Source Location
- Repository: datasets
- File:
src/datasets/packaged_modules/webdataset/_tenbin.py - Lines: 1-285
Key Functions
# Buffer-level operations (highest-level in-memory)
def encode_buffer(l, infos=None):
"""Encode a list of arrays into a single byte array."""
def decode_buffer(buf, infos=False):
"""Decode a byte array into a list of arrays."""
# List-level operations (array <-> header+data pairs)
def encode_list(l, infos=None):
"""Given a list of arrays, encode them into a list of byte arrays."""
def decode_list(l, infos=False):
"""Given a list of byte arrays, decode them into arrays."""
# Chunk-level operations (byte alignment and magic numbers)
def encode_chunks(l):
"""Encode a list of chunks into a single byte array, with lengths and magics."""
def decode_chunks(buf):
"""Decode a byte array into a list of chunks."""
# Stream I/O
def write(stream, l, infos=None):
"""Write a list of arrays to a stream, with magics, length, and padding."""
def read(stream, n=sys.maxsize, infos=False):
"""Read a list of arrays from a stream, with magics, length, and padding."""
# File I/O
def save(fname, *args, infos=None, nocheck=False):
"""Save a list of arrays to a file, with magics, length, and padding."""
def load(fname, infos=False, nocheck=False):
"""Read a list of arrays from a file, with magics, length, and padding."""
Internal Helper Functions
def bytelen(a):
"""Determine the length of a in bytes."""
def bytedata(a):
"""Return the raw data corresponding to a."""
def check_acceptable_input_type(data, allow64):
"""Check that the data has an acceptable type for tensor encoding."""
def str64(s):
"""Convert a string to an int64."""
def unstr64(i):
"""Convert an int64 to a string."""
def check_infos(data, infos, required_infos=None):
"""Verify the info strings."""
def encode_header(a, info=""):
"""Encode an array header as a byte array."""
def decode_header(h):
"""Decode a byte array into an array header."""
def roundup(n, k=64):
"""Round up to the next multiple of 64."""
def write_chunk(stream, buf):
"""Write a byte chunk to the stream with magics, length, and padding."""
def read_chunk(stream):
"""Read a byte chunk from a stream with magics, length, and padding."""
I/O Contract
Binary Format Structure
| Component | Size | Description |
|---|---|---|
| Magic number | 8 bytes | ASCII string "~TenBin~" encoded as int64.
|
| Length | 8 bytes | Size of the following data in bytes (int64). |
| Data | Variable | Payload bytes, padded to a multiple of 64 bytes. |
Array Header Structure
| Field | Size | Description |
|---|---|---|
| dtype | 8 bytes | Short dtype code (e.g., "f4" for float32) encoded as int64.
|
| info | 8 bytes | Optional 8-character info/name string encoded as int64. |
| ndim | 8 bytes | Number of dimensions (int64). |
| shape | 8 bytes each | One int64 per dimension. |
Supported Dtypes
| Long Name | Short Code |
|---|---|
| float16 | f2 |
| float32 | f4 |
| float64 | f8 |
| int8 | i1 |
| int16 | i2 |
| int32 | i4 |
| int64 | i8 |
| uint8 | u1 |
| uint16 | u2 |
| uint32 | u4 |
| uint64 | u8 |
Usage Examples
Encoding and Decoding Arrays in Memory
import numpy as np
from datasets.packaged_modules.webdataset._tenbin import encode_buffer, decode_buffer
# Encode arrays to binary
arrays = [np.array([1.0, 2.0, 3.0], dtype="float32"), np.zeros((2, 3), dtype="int32")]
buf = encode_buffer(arrays)
# Decode binary back to arrays
decoded = decode_buffer(buf)
print(decoded[0]) # [1. 2. 3.]
print(decoded[1].shape) # (2, 3)
Saving and Loading from Files
import numpy as np
from datasets.packaged_modules.webdataset._tenbin import save, load
a = np.random.randn(10, 20).astype("float32")
b = np.arange(100, dtype="int64")
# Save to a .ten file
save("data.ten", a, b)
# Load from a .ten file
arrays = load("data.ten")
print(arrays[0].shape) # (10, 20)
print(arrays[1].shape) # (100,)