Implementation:Datajuicer Data juicer Check S3 Integration

Knowledge Sources	Datajuicer_Data_juicer
Domains	Cloud Storage, Integration Testing, S3
Last Updated	2026-02-14 16:00 GMT

Overview

Standalone integration test script that validates S3 data loading and exporting capabilities across multiple backends (HuggingFace datasets, Ray datasets) and access modes (public/anonymous, private/authenticated).

Description

The check_s3_integration script provides comprehensive S3 integration testing organized into three test categories:

Low-Level API Tests (Tests 1-4):

• test_s3_load_public_file -- Loads a public JSONL file from S3 using HuggingFace datasets with anonymous access via storage_options={"anon": True}.
• test_s3_load_private_file -- Loads a private JSONL file using HuggingFace datasets with AWS credentials obtained via get_aws_credentials().
• test_ray_s3_load_public_file -- Loads a public file using Ray datasets with a PyArrow S3 filesystem created via create_pyarrow_s3_filesystem().
• test_ray_s3_load_private_file -- Loads a private file using Ray datasets with credential-configured PyArrow filesystem.

Load Strategy Integration Tests (Tests 5-8):

• test_strategy_s3_load_public_file -- Tests DefaultS3DataLoadStrategy for public bucket access via the DataLoadStrategyRegistry.
• test_strategy_s3_load_private_file -- Tests DefaultS3DataLoadStrategy for private bucket access with credentials.
• test_strategy_ray_s3_load_public_file -- Tests RayS3DataLoadStrategy for public bucket access with region configuration.
• test_strategy_ray_s3_load_private_file -- Tests RayS3DataLoadStrategy for private bucket access.

Export/Upload Tests (Tests 9-10):

• test_s3_export_private_file -- Tests the HuggingFace Exporter class for uploading datasets to private S3 buckets, with verification by loading the exported data back.
• test_ray_s3_export_private_file -- Tests the RayExporter class for uploading Ray datasets to private S3 with verification.

Each test function:

• Handles missing credentials gracefully by skipping instead of failing.
• Auto-detects file format from extension (json, jsonl, csv, parquet, text).
• Reports detailed error messages with common issue diagnostics.
• Returns boolean success/failure for summary reporting.

The main() function runs all tests sequentially and prints a summary of pass/fail results.

Usage

Run this script to verify that S3 integration is correctly configured before deploying Data-Juicer in cloud environments. It validates credential configuration, network connectivity, and compatibility across all supported execution modes.

Code Reference

Source Location

• Repository: Datajuicer_Data_juicer
• File: tools/check_s3_integration.py

Signature

def test_s3_load_public_file() -> bool: ...
def test_s3_load_private_file(s3_path: str = None) -> bool: ...
def test_ray_s3_load_public_file() -> bool: ...
def test_ray_s3_load_private_file(s3_path: str = None) -> bool: ...
def test_strategy_s3_load_public_file() -> bool: ...
def test_strategy_s3_load_private_file(s3_path: str = None) -> bool: ...
def test_strategy_ray_s3_load_public_file() -> bool: ...
def test_strategy_ray_s3_load_private_file(s3_path: str = None) -> bool: ...
def test_s3_export_private_file() -> bool: ...
def test_ray_s3_export_private_file() -> bool: ...
def main() -> int: ...

Import

# Typically run as a standalone script
# python tools/check_s3_integration.py

# Or import individual test functions
from tools.check_s3_integration import (
    test_s3_load_public_file,
    test_s3_load_private_file
)

I/O Contract

Inputs

Name	Type	Required	Description
s3_path	str	No	Custom S3 path for private file tests. Defaults to internal test buckets.
AWS_ACCESS_KEY_ID	env var	No	AWS access key for private bucket tests. Tests skip gracefully if missing.
AWS_SECRET_ACCESS_KEY	env var	No	AWS secret key for private bucket tests.
AWS_SESSION_TOKEN	env var	No	Optional AWS session token.

Outputs

Name	Type	Description
return code	int	0 if all tests pass, 1 if any test fails.
console output	str	Detailed test results with pass/fail status and error diagnostics.

Usage Examples

# Run all S3 integration tests
# Requires: pip install s3fs datasets ray pyarrow
python tools/check_s3_integration.py

# With credentials for private bucket tests
export AWS_ACCESS_KEY_ID="your-key"
export AWS_SECRET_ACCESS_KEY="your-secret"
python tools/check_s3_integration.py

# Programmatic usage
from tools.check_s3_integration import (
    test_s3_load_public_file,
    test_strategy_s3_load_private_file
)

# Test public S3 access
assert test_s3_load_public_file()

# Test private S3 access with custom path
assert test_strategy_s3_load_private_file(
    s3_path="s3://my-bucket/my-data.jsonl"
)

Related Pages

• Datajuicer_Data_juicer_File_Utils -- File utility functions with remote path detection used by S3 strategies