Implementation:Datajuicer Data juicer NLTK Utils

Knowledge Sources	Datajuicer_Data_juicer
Domains	NLP, Compatibility
Last Updated	2026-02-14 16:00 GMT

Overview

Utilities for handling NLTK-specific operations including resource downloading, pickle security patching, and path mapping to resolve cross-version compatibility issues.

Description

The nltk_utils module solves compatibility problems that arise between different NLTK versions, particularly regarding resource naming and pickle security restrictions. Key components include:

Resource Path Mappings:

• path_mappings -- Dictionary mapping problematic NLTK resource paths to their correct equivalents (e.g., taggers/averaged_perceptron_tagger_eng maps to taggers/averaged_perceptron_tagger/english.pickle).
• resource_fallbacks -- Dictionary mapping new package names to legacy equivalents (e.g., punkt_tab falls back to punkt).

Resource Management:

• ensure_nltk_resource -- Ensures a specific NLTK resource is available by first checking for known problematic paths and remapping them, proactively downloading the fallback package, and retrying with multiple download strategies. Includes special handling for the averaged_perceptron_tagger resource.
• clean_nltk_cache -- Cleans NLTK data directories, supporting selective package cleaning or complete resets across all NLTK data directories.

Security Patching:

• patch_nltk_pickle_security -- Patches NLTK 3.9+ strict pickle security restrictions that prevent loading some models. Replaces restricted_pickle_load with a more permissive version and adds required classes (PunktSentenceTokenizer, PunktParameters, etc.) to the allowed pickle classes list.

File System Aliasing:

• create_physical_resource_alias -- Creates filesystem aliases (symlinks, hard links, or copies) for NLTK resources that have path mismatches between versions.
• setup_resource_aliases -- Pre-creates common aliases, particularly for the averaged_perceptron_tagger to averaged_perceptron_tagger_eng mapping.

Usage

Use this module before any NLTK operations to ensure resources are available and security patches are applied. It is called by prepare_nltk_model and prepare_nltk_pos_tagger in model_utils.

Code Reference

Source Location

• Repository: Datajuicer_Data_juicer
• File: data_juicer/utils/nltk_utils.py

Signature

def ensure_nltk_resource(resource_path: str,
                         fallback_package: str = None) -> bool: ...
def clean_nltk_cache(packages: list = None,
                     complete_reset: bool = False) -> None: ...
def patch_nltk_pickle_security() -> bool: ...
def create_physical_resource_alias(source_path: str,
                                   alias_path: str) -> bool: ...
def setup_resource_aliases() -> bool: ...

Import

from data_juicer.utils.nltk_utils import (
    ensure_nltk_resource, patch_nltk_pickle_security,
    clean_nltk_cache, setup_resource_aliases
)

I/O Contract

Inputs

Name	Type	Required	Description
resource_path	str	Yes	NLTK resource path to verify (e.g., "tokenizers/punkt/english.pickle").
fallback_package	str	No	NLTK package to download if resource is missing (e.g., "punkt").
packages	list	No	List of NLTK package names to clean. None cleans all.
complete_reset	bool	No	If True, deletes all NLTK data directories entirely.
source_path	str	Yes	Full filesystem path to the source resource file.
alias_path	str	Yes	Full filesystem path where the alias should be created.

Outputs

Name	Type	Description
success	bool	Whether the resource was found/ensured, patch was applied, or alias was created.

Usage Examples

from data_juicer.utils.nltk_utils import (
    patch_nltk_pickle_security, ensure_nltk_resource,
    setup_resource_aliases
)

# Patch pickle security (call once at initialization)
patch_nltk_pickle_security()

# Ensure punkt tokenizer is available
available = ensure_nltk_resource(
    "tokenizers/punkt/english.pickle",
    fallback_package="punkt"
)
if available:
    import nltk
    tokenizer = nltk.data.load("tokenizers/punkt/english.pickle")

# Set up aliases for cross-version compatibility
setup_resource_aliases()

# Clean specific NLTK caches
from data_juicer.utils.nltk_utils import clean_nltk_cache
clean_nltk_cache(packages=["punkt", "averaged_perceptron_tagger"])

Related Pages

• Datajuicer_Data_juicer_Model_Utils -- Uses nltk_utils in prepare_nltk_model and prepare_nltk_pos_tagger