Implementation:Datajuicer Data juicer File Utils

Knowledge Sources	Datajuicer_Data_juicer
Domains	Data Processing, File Management
Last Updated	2026-02-14 16:00 GMT

Overview

Comprehensive file system utility module providing functions for path manipulation, file discovery, size formatting, remote path detection, async file reading, and dataset I/O operations.

Description

The file_utils module is a foundational utility used extensively across the Data-Juicer framework. It provides:

Size and Formatting:

• Sizes -- Class with byte unit constants (KiB, MiB, GiB, TiB).
• byte_size_to_size_str -- Converts byte counts to human-readable strings (e.g., "2.50 GiB").

File Discovery and Path Utilities:

• find_files_with_suffix -- Recursively traverses a directory to find files with specified suffixes, with support for compressed file extensions (e.g., ".jsonl.zst").
• is_remote_path -- Detects S3, HTTP, HTTPS, GS, and HDFS paths.
• is_absolute_path -- Checks if a path is absolute, including remote paths.
• add_suffix_to_filename -- Inserts a suffix before the file extension.
• create_directory_if_not_exists -- Process-safe directory creation.
• get_all_files_paths_under -- Lists all files under a directory with optional recursion.

Multimodal Data Management:

• transfer_data_dir -- Creates output directories for newly generated multimodal data following the __dj__produced_data__/{op_name} pattern.
• transfer_filename -- Generates unique output filenames with hash-based deduplication, supporting custom save directories and the DJ_PRODUCED_DATA_DIR environment variable.
• copy_data -- Copies files between directories.

Dataset I/O:

• single_partition_write_with_filename -- Writes DataFrame partitions to JSONL or Parquet files grouped by a "filename" column.
• read_single_partition -- Reads JSONL, JSON, or Parquet files into a pandas DataFrame with optional column selection and filename tracking.

Async Operations:

• follow_read -- Async generator that tails a file for new content (similar to tail -f).
• download_file -- Async HTTP file downloader using aiohttp with configurable timeout.

Usage

Use this module for any file system operation within the Data-Juicer framework, including path resolution, dataset export, multimodal data directory management, and remote file detection.

Code Reference

Source Location

• Repository: Datajuicer_Data_juicer
• File: data_juicer/utils/file_utils.py

Signature

class Sizes:
    KiB = 2**10
    MiB = 2**20
    GiB = 2**30
    TiB = 2**40

def byte_size_to_size_str(byte_size: int) -> str: ...
async def follow_read(logfile_path: str,
                      skip_existing_content: bool = False) -> AsyncGenerator: ...
def find_files_with_suffix(path, suffixes=None) -> Dict[str, List[str]]: ...
def is_remote_path(path: str) -> bool: ...
def is_absolute_path(path) -> bool: ...
def add_suffix_to_filename(filename, suffix) -> str: ...
def transfer_filename(original_filepath, op_name,
                      save_dir=None, **op_kwargs) -> str: ...
def read_single_partition(files, filetype="jsonl", add_filename=False,
                          input_meta=None, columns=None) -> pd.DataFrame: ...
async def download_file(session, url, save_path=None,
                        return_content=False, timeout=300): ...

Import

from data_juicer.utils.file_utils import (
    find_files_with_suffix, is_remote_path, transfer_filename,
    add_suffix_to_filename, byte_size_to_size_str
)

I/O Contract

Inputs

Name	Type	Required	Description
path	Union[str, Path]	Yes	File or directory path to search or manipulate.
suffixes	Union[str, List[str]]	No	File suffixes to filter by (e.g., ".txt", [".jsonl", ".parquet"]).
original_filepath	Union[str, Path]	Yes	Original file path to transform for output.
op_name	str	Yes	Operator name used in directory naming for produced data.
save_dir	str	No	Custom save directory. Overrides default directory logic.

Outputs

Name	Type	Description
file_dict	Dict[str, List[str]]	Dictionary mapping file suffixes to lists of matching file paths.
new_filepath	str	Transformed output file path with hash-based unique naming.
df	pd.DataFrame	DataFrame read from JSONL/JSON/Parquet files with sorted columns.

Usage Examples

from data_juicer.utils.file_utils import (
    find_files_with_suffix, is_remote_path, transfer_filename,
    byte_size_to_size_str
)

# Find all JSONL files in a directory
file_dict = find_files_with_suffix("/data/input", suffixes=[".jsonl"])
print(file_dict)  # {".jsonl": ["/data/input/a.jsonl", ...]}

# Check if a path is remote
is_remote_path("s3://bucket/data.jsonl")  # True
is_remote_path("/local/data.jsonl")       # False

# Generate a unique output filename
new_path = transfer_filename(
    "/data/images/photo.jpg", "resize_mapper",
    width=256, height=256
)
# Result: /data/images/__dj__produced_data__/resize_mapper/photo__dj_hash_#abc123#.jpg

# Human-readable file size
print(byte_size_to_size_str(1536 * 1024 * 1024))  # "1.50 GiB"

Related Pages

• Datajuicer_Data_juicer_Compress_Utils -- Compression utilities for cache file management
• Datajuicer_Data_juicer_Multimodal_Utils -- Uses file_utils for multimodal data path management