Implementation:Datajuicer Data juicer DataValidator

Knowledge Sources	Datajuicer_Data_juicer
Domains	Data Validation, Schema Checking, Quality Assurance
Last Updated	2026-02-14 16:00 GMT

Overview

Pluggable data validation framework with an abstract base class, a decorator-based registry, and concrete validators for conversation formats, code data, and required field checks.

Description

This module defines the validation infrastructure for checking dataset schemas at load time. The architecture consists of:

Core Classes:

DataValidator (ABC) -- Abstract base class defining the `validate(dataset)` interface. Validates that the input is a DJDataset instance.
DataValidationError -- Custom exception for validation failures.
DataValidatorRegistry -- Registry using a `@register(validator_type)` class decorator to map string names to validator classes.

Conversation Validators:

BaseConversationValidator -- Abstract class extending DataValidator with configurable min_turns, max_turns, and sample_size. Iterates over dataset samples calling `validate_conversation()`.
SwiftMessagesValidator (type: "swift_messages") -- Validates Swift Messages format: checks for `messages` array, valid roles (system/user/assistant), and required content fields.
DataJuicerFormatValidator (type: "dj_conversation") -- Validates Data-Juicer default format: checks for instruction/query/response fields and optional history as [query, response] pairs.

Other Validators:

CodeDataValidator (type: "code") -- Validates code data with required columns and supported languages (placeholder implementation).
RequiredFieldsValidator (type: "required_fields") -- Checks that specified fields exist, validates their types using a string-to-type mapping (TYPE_NAME_MAPPING), and enforces missing value ratio thresholds.

Usage

Configure validators in YAML configs under the `validators` key. They run automatically during dataset loading to catch schema issues before expensive processing begins.

Code Reference

Source Location

Repository: Datajuicer_Data_juicer
File: data_juicer/core/data/data_validator.py
Lines: 1-368

Signature

class DataValidator(ABC):
    def __init__(self, config: Dict): ...
    @abstractmethod
    def validate(self, dataset: DJDataset) -> None: ...

class DataValidatorRegistry:
    @classmethod
    def register(cls, validator_type: str): ...
    @classmethod
    def get_validator(cls, validator_type: str) -> Optional[Type[DataValidator]]: ...

@DataValidatorRegistry.register("swift_messages")
class SwiftMessagesValidator(BaseConversationValidator): ...

@DataValidatorRegistry.register("dj_conversation")
class DataJuicerFormatValidator(BaseConversationValidator): ...

@DataValidatorRegistry.register("required_fields")
class RequiredFieldsValidator(DataValidator): ...

Import

from data_juicer.core.data.data_validator import (
    DataValidator,
    DataValidatorRegistry,
    DataValidationError,
    SwiftMessagesValidator,
    DataJuicerFormatValidator,
    RequiredFieldsValidator,
)

I/O Contract

Inputs

Name	Type	Required	Description
config	Dict	Yes	Configuration dictionary with validator-specific parameters (min_turns, max_turns, sample_size, required_fields, field_types, allow_missing)
dataset	DJDataset	Yes	The dataset to validate (NestedDataset or RayDataset)

Outputs

Name	Type	Description
None	None	Raises DataValidationError if validation fails; returns None on success

Usage Examples

# Using RequiredFieldsValidator programmatically:
from data_juicer.core.data.data_validator import RequiredFieldsValidator

validator = RequiredFieldsValidator({
    "required_fields": ["text", "label"],
    "field_types": {"text": "str", "label": "int"},
    "allow_missing": 0.05,
    "sample_size": 200,
})
validator.validate(dataset)  # Raises DataValidationError if checks fail

# In YAML config:
# validators:
#   - type: 'required_fields'
#     required_fields: ["text"]
#     field_types:
#       text: 'str'
#   - type: 'swift_messages'
#     min_turns: 2
#     max_turns: 50

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment