Environment:Google research Deduplicate text datasets Python HuggingFace Environment

Knowledge Sources	deduplicate-text-datasets HuggingFace Datasets
Domains	Infrastructure, NLP, Text_Deduplication
Last Updated	2026-02-14 21:00 GMT

Overview

Python 3 environment with HuggingFace `datasets`, `transformers`, `tqdm`, and `numpy` for HuggingFace-based dataset loading and serialization to flat binary format.

Description

This environment provides the Python runtime needed for the HuggingFace dataset loading path (`load_dataset_hf.py`). It uses the HuggingFace `datasets` library to load datasets by name or from local files (text, JSON, CSV), and optionally tokenizes them with the GPT-2 tokenizer from the `transformers` library. The output is a flat binary file compatible with the Rust suffix array tools.

Usage

Use this environment when loading datasets from HuggingFace Hub or local files instead of TensorFlow Datasets. This is the alternative to the TFDS environment for users who prefer the HuggingFace ecosystem. It is the mandatory prerequisite for running the `Load_Dataset_HF` implementation.

System Requirements

Category	Requirement	Notes
OS	Linux (Ubuntu recommended)	macOS may also work
Hardware	CPU (no GPU required)	Dataset loading is CPU-bound
RAM	Proportional to dataset size	HuggingFace `datasets` uses memory-mapped files for large datasets
Disk	2x dataset size minimum	Downloaded dataset + serialized binary output

Dependencies

System Packages

• Python 3 runtime

Python Packages

• `datasets` (HuggingFace datasets library)
• `transformers` (for GPT2Tokenizer)
• `numpy`
• `tqdm`

Credentials

No credentials required for public datasets. For private HuggingFace Hub datasets:

• `HF_TOKEN`: HuggingFace API token with read access.

Quick Install

# Install Python dependencies for HuggingFace workflow
pip3 install datasets transformers numpy tqdm

Code Evidence

HuggingFace imports from `scripts/load_dataset_hf.py:14-19`:

import datasets
import os
import struct
import numpy as np
from transformers import GPT2Tokenizer
from tqdm import tqdm

Dataset loading logic supporting both named datasets and local files from `scripts/load_dataset_hf.py:51-56`:

if dataset_name in FILE_EXTENSIONS:
    assert data_dir is not None
    data_files = glob.glob(f"{data_dir}/*.{FILE_EXTENSIONS[dataset_name]}")
    ds = datasets.load_dataset(dataset_name, subset, data_files=data_files, split=split)
else:
    ds = datasets.load_dataset(dataset_name, subset, split=split)

Type assertion from `scripts/load_dataset_hf.py:57`:

assert isinstance(ds, datasets.Dataset), "This is not a HF-dataset. It might be a DatasetDict. Try passing `split`?"

Common Errors

Error Message	Cause	Solution
`ModuleNotFoundError: No module named 'datasets'`	HuggingFace datasets not installed	`pip3 install datasets`
`AssertionError: This is not a HF-dataset. It might be a DatasetDict. Try passing 'split'?`	Missing `--split` argument	Pass `--split train` or `--split test` to specify which split to load
`AssertionError` on `data_dir is not None`	Loading local files without specifying data directory	Pass `--data_dir /path/to/files` when using text/json/csv format

Compatibility Notes

• Local file formats: Supports `text` (.txt), `json` (.jsonl), and `csv` (.csv) via the `FILE_EXTENSIONS` mapping.
• Tokenization: Optional `--tokenize` flag uses GPT-2 tokenizer only (unlike the TFDS script which also supports T5).
• Parallel tokenization: Supports `--num_workers` for parallel `datasets.map()` processing.

Related Pages

• Implementation:Google_research_Deduplicate_text_datasets_Load_Dataset_HF