Environment:LLMBook zh LLMBook zh github io Data Processing Environment

Knowledge Sources	LLMBook-zh NLTK
Domains	Data_Processing, NLP
Last Updated	2026-02-08 04:30 GMT

Overview

Python environment with NLTK, regex, and text processing utilities for LLM data preprocessing (quality filtering, deduplication, privacy filtering, BPE tokenization).

Description

This environment provides the standard Python libraries needed for the data preprocessing pipeline (Chapter 4). It includes NLTK for n-gram generation used in deduplication, re for regex-based pattern matching in privacy filtering and text processing, string for punctuation constants, and collections.Counter for frequency counting in BPE tokenization. Unlike the training environments, this stack does not require GPU access and runs on CPU.

Usage

Use this environment for all data preprocessing workflows: language quality filtering, n-gram deduplication, PII masking, and BPE tokenizer training. This is a prerequisite before any training workflow can begin.

System Requirements

Category	Requirement	Notes
OS	Linux, macOS, or Windows	Cross-platform support
Hardware	CPU	No GPU required for data preprocessing
Python	Python >= 3.7	Standard CPython
Disk	Varies	Depends on dataset size

Dependencies

Python Packages

`nltk` >= 3.7
`json` (stdlib)
`re` (stdlib)
`string` (stdlib)
`collections` (stdlib)

Additional Resources

fastText language model: Required by `LangIdentifier` for quality filtering (`code/4.1 质量过滤.py`)
Custom utility modules: `utils.evaluator.LangIdentifier`, `utils.rules.regex.REGEX_IDCARD`, `utils.cleaner.cleaner_base.CleanerBase`

Credentials

No credentials required.

Quick Install

# Install NLTK
pip install nltk

# Download NLTK data (if needed for tokenizers)
python -c "import nltk; nltk.download('punkt')"

Code Evidence

NLTK n-gram usage from `code/4.2 去重.py:3`:

from nltk.util import ngrams

Regex and string utilities from `code/4.2 去重.py:1-2`:

import string
import re

Collections Counter for BPE from `code/4.4 BPE分词.py` (used for frequency counting of byte pairs).

Custom utility imports from `code/4.1 质量过滤.py:1`:

from utils.evaluator import LangIdentifier

Common Errors

Error Message	Cause	Solution
`ModuleNotFoundError: No module named 'nltk'`	NLTK not installed	`pip install nltk`
`ModuleNotFoundError: utils.evaluator`	Custom utility module not in path	Ensure the project root with `utils/` package is in `PYTHONPATH`
`LookupError: NLTK data not found`	NLTK resources not downloaded	Run `nltk.download('punkt')` or other required resources

Compatibility Notes

CPU-only: All data preprocessing scripts run on CPU. No GPU required.
Custom Modules: The `utils` package (LangIdentifier, regex patterns, CleanerBase) is part of the textbook's companion code and must be available in the Python path.
fastText: Language identification requires a pre-trained fastText model file.

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment