Implementation:Datajuicer Data juicer OpSearch

Knowledge Sources	Datajuicer_Data_juicer
Domains	Data Processing, Operator Discovery
Last Updated	2026-02-14 16:00 GMT

Overview

Operator discovery and search system that indexes all registered Data-Juicer operators with their metadata and provides filtering by operator type and tags.

Description

The op_search module implements a programmatic introspection engine for the entire Data-Juicer operator ecosystem. It consists of two main classes:

• OPRecord -- A record class that stores comprehensive metadata about a single operator, including its name, type (mapper, filter, deduplicator, etc.), description, tags (modality, resource, model), signature, parameter descriptions, source path, and test path. Tags are derived by analyzing the operator class source code and inheritance chain.

• OPSearcher -- A search engine that scans all registered operators (from the OPERATORS and FORMATTERS registries), creates OPRecord instances for each, and supports search queries filtered by operator type and tags with configurable match-all or match-any semantics.

Tag analysis is performed through several dedicated functions:

• analyze_modality_tag examines source code for self.text_key, self.image_key, etc. to determine modality (text, image, audio, video, multimodal).
• analyze_resource_tag reads the _accelerator class attribute to determine CPU or GPU usage.
• analyze_model_tags uses regex to find model_type arguments in prepare_model calls to identify API, vLLM, or HuggingFace model dependencies.
• analyze_tag_with_inheritance traverses the MRO chain (up to 3 levels) to resolve tags from parent classes.

Usage

Use this module when you need to programmatically discover operators by their capabilities, generate documentation, or build tooling (such as the MCP server) that needs to enumerate available operators with metadata. It can also be used as a standalone CLI tool.

Code Reference

Source Location

• Repository: Datajuicer_Data_juicer
• File: data_juicer/tools/op_search.py

Signature

class OPRecord:
    def __init__(self, name: str, op_cls: type, op_type: str = None): ...
    def to_dict(self) -> dict: ...

class OPSearcher:
    def __init__(self, specified_op_list: Optional[List[str]] = None,
                 include_formatter: bool = False): ...
    def search(self, tags: Optional[List[str]] = None,
               op_type: Optional[str] = None,
               match_all: bool = True) -> List[Dict]: ...

def analyze_modality_tag(code: str, op_prefix: str) -> list: ...
def analyze_resource_tag(cls: type) -> list: ...
def analyze_model_tags(cls: type) -> list: ...
def analyze_tag_from_cls(op_cls: type, op_name: str) -> list: ...

Import

from data_juicer.tools.op_search import OPSearcher, OPRecord

I/O Contract

Inputs

Name	Type	Required	Description
specified_op_list	List[str]	No	List of operator names to scan. If None, scans all registered operators.
include_formatter	bool	No	Whether to include formatter operators in the scan. Default False.
tags	List[str]	No	Tags to filter by (e.g. "gpu", "text", "api").
op_type	str	No	Operator type to filter by (e.g. "mapper", "filter", "deduplicator").
match_all	bool	No	If True, require all tags to match. If False, match any tag.

Outputs

Name	Type	Description
results	List[Dict]	List of operator metadata dicts with keys: type, name, desc, tags, sig, param_desc, param_desc_map, source_path, test_path.
all_ops	Dict[str, OPRecord]	Dictionary mapping operator names to their OPRecord instances.

Usage Examples

from data_juicer.tools.op_search import OPSearcher

# Scan all operators including formatters
searcher = OPSearcher(include_formatter=True)

# Find all GPU-accelerated mappers
results = searcher.search(tags=["gpu"], op_type="mapper")
for op in results:
    print(f"[{op['type']}] {op['name']} - Tags: {op['tags']}")

# Find any operator that works with images or video
results = searcher.search(tags=["image", "video"], match_all=False)

# Access a specific operator record
record = searcher.all_ops["nlpaug_en_mapper"]
print(record.source_path, record.test_path)

Related Pages

• Datajuicer_Data_juicer_Model_Utils -- Model preparation functions referenced by model tag analysis
• Datajuicer_Data_juicer_Multimodal_Utils -- Multimodal utilities that define modality-specific keys