Implementation:Huggingface Datasets Dill Pickler

Source	src/datasets/utils/_dill.py
Domain(s)	Serialization, Caching
Last Updated	2026-02-14

Overview

Description

The Dill Pickler module extends Python's dill serialization library to produce deterministic, content-addressed pickle output. The standard dill.Pickler does not guarantee consistent byte-level output for semantically identical objects -- dictionary key ordering, set element ordering, and string memoization can all vary between runs. This module solves that problem by subclassing dill.Pickler and overriding three critical methods:

• save -- Intercepts serialization of objects from third-party libraries (regex, spaCy, tiktoken, PyTorch, Transformers) and registers custom reducers so they serialize consistently. It also unwraps torch.compile-ed modules and functions to their originals.
• _batch_setitems -- Sorts dictionary items before writing them to the pickle stream, ensuring that dictionaries with the same content always produce the same bytes regardless of insertion order. Falls back to hash-based sorting for unorderable keys.
• memoize -- Disables memoization for strings, since two identical string values can have different Python object IDs, which would break determinism.

The module also provides convenience functions dump and dumps that mirror the standard pickle API but use the custom Pickler class. Additionally, it registers custom serializers for set (sorted before pickling) and CodeType (filename and line number normalized) to further ensure reproducible output.

This deterministic behavior is essential for the datasets library's fingerprinting system, which hashes processing functions and their arguments to build cache keys. Without deterministic serialization, cache lookups would fail even when the same transform is applied.

Usage

Use this module when you need deterministic serialization of Python objects for caching or fingerprinting purposes. The Pickler class is used internally by datasets.fingerprint.Hasher to produce consistent hashes. The dump and dumps functions serve as drop-in replacements for pickle.dump and pickle.dumps.

Code Reference

Source Location

src/datasets/utils/_dill.py (467 lines)

Signature

class Pickler(dill.Pickler):
    dispatch = dill._dill.MetaCatchingDict(dill.Pickler.dispatch.copy())
    _legacy_no_dict_keys_sorting = False

    def save(self, obj, save_persistent_id=True): ...
    def _batch_setitems(self, items, *args, **kwargs): ...
    def memoize(self, obj): ...

def dump(obj, file): ...
def dumps(obj): ...

Import

from datasets.utils._dill import Pickler, dump, dumps

I/O Contract

Inputs

Name	Type	Description
obj	Any	The Python object to serialize
file	file-like object	A writable binary file (for dump)

Outputs

Name	Type	Description
pickle bytes	bytes	Deterministic pickle byte stream (from dumps)
(side effect)	written to file	Pickle bytes written to the file object (from dump)

Usage Examples

Serializing an object to bytes deterministically:

from datasets.utils._dill import dumps

# Two dicts with the same content but different insertion order
# will produce identical pickle bytes
d1 = {"b": 2, "a": 1}
d2 = {"a": 1, "b": 2}

assert dumps(d1) == dumps(d2)

Serializing to a file:

from datasets.utils._dill import dump

with open("cache.pkl", "wb") as f:
    dump({"key": "value", "data": [1, 2, 3]}, f)

Using the Pickler class directly:

from io import BytesIO
from datasets.utils._dill import Pickler

buffer = BytesIO()
pickler = Pickler(buffer, recurse=True)
pickler.dump(my_transform_function)
deterministic_bytes = buffer.getvalue()

Related Pages

• Principle: Deterministic Serialization -- The design principle requiring that semantically identical objects always produce the same serialized bytes, enabling reliable cache lookups and fingerprinting.