Implementation:Datajuicer Data juicer Config All Yaml

Knowledge Sources	Datajuicer_Data_juicer
Domains	Configuration, Reference Documentation, YAML
Last Updated	2026-02-14 16:00 GMT

Overview

Comprehensive reference configuration file that documents every global parameter, operator, and their arguments available in Data-Juicer, serving as both a template and a complete example config.

Description

This YAML file acts as the authoritative reference for all configurable options in Data-Juicer. It defines:

Global Settings:

Project metadata: project_name, dataset_path, export_path, export_type, export_shard_size
Dataset configuration: Inline dataset specification with type, weight, and path for multi-source mixing
Validators: Pluggable dataset validators (e.g., required_fields) applied at load time
Parallelism: auto_op_parallelism, np (number of processes), adaptive_batch_size
Caching: use_cache, ds_cache_dir, cache_compress, use_checkpoint, temp_dir
Monitoring: open_monitor, open_tracer, op_list_to_trace, trace_num, trace_keys
Multimodal keys: image_key, audio_key, video_key, and their special tokens
Distributed processing: executor_type (default/ray), ray_address, partition configuration
Analysis: percentiles for dataset distribution analysis

Process List: An exhaustive list of every built-in operator with their full parameter sets and inline documentation comments, organized by type:

Mappers (text transformation, LLM-based generation, multimodal processing)
Filters (text quality, image quality, language identification, LLM-based scoring)
Deduplicators (exact match, MinHash, SimHash, image/video deduplication)
Selectors (frequency-based, random, range, topk)
Groupers (key-based, naive)
Aggregators (entity, meta, nested model)

Usage

Use this file as a reference to discover available operators and understand what parameters each one accepts. Copy and modify relevant sections to create custom processing configurations.

Code Reference

Source Location

Repository: Datajuicer_Data_juicer
File: data_juicer/config/config_all.yaml
Lines: 1-1213

Signature

# Top-level structure
project_name: 'all'
dataset_path: '/path/to/your/dataset'
export_path: '/path/to/result/dataset.jsonl'
np: 4
text_keys: 'text'

# Multimodal keys
image_key: 'images'
audio_key: 'audios'
video_key: 'videos'

# Distributed processing
executor_type: default
ray_address: auto

# Processing pipeline
process:
  - operator_name:
      param1: value1
      param2: value2

Import

# Load the config file using Data-Juicer's config system:
from data_juicer.config import init_configs
cfg = init_configs(args=["--config", "data_juicer/config/config_all.yaml"])

I/O Contract

Inputs

Name	Type	Required	Description
N/A	N/A	N/A	This is a reference configuration file; it serves as input to the Data-Juicer config parser

Outputs

Name	Type	Description
Parsed configuration	Namespace	When loaded by init_configs, produces a fully-resolved configuration namespace with all parameters

Usage Examples

# Minimal config derived from config_all.yaml:
project_name: 'my_project'
dataset_path: './data/input.jsonl'
export_path: './data/output.jsonl'
np: 8

process:
  - language_id_score_filter:
      lang: 'en'
      min_score: 0.8
  - text_length_filter:
      min_len: 100
      max_len: 10000
  - document_minhash_deduplicator:
      tokenization: 'space'
      jaccard_threshold: 0.7

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment