Implementation:Datajuicer Data juicer TextEntityDependencyFilter
| Knowledge Sources | |
|---|---|
| Domains | Data_Quality, Filtering |
| Last Updated | 2026-02-14 16:00 GMT |
Overview
Concrete tool for filtering data samples based on entity dependency relationships provided by Data-Juicer.
Description
TextEntityDependencyFilter is a filter operator that keeps samples where detected entities have sufficient dependency edges in the parse tree. It extends Filter and uses the two-phase compute_stats/process pattern. It uses a spaCy NLP model (English or Chinese) to identify entities (nouns, proper nouns, pronouns) by POS and tag, then counts each entity's dependency edges in the parse tree. The counts are cached under num_dependency_edges. Samples are kept using 'any' (at least one entity meets the threshold) or 'all' (every entity must meet it) strategy against min_dependency_num. Samples with no detected entities are omitted.
Usage
Import when filtering based on entity dependency structure. Configure in YAML or Python.
Code Reference
Source Location
- Repository: Datajuicer_Data_juicer
- File: data_juicer/ops/filter/text_entity_dependency_filter.py
Signature
@OPERATORS.register_module("text_entity_dependency_filter")
class TextEntityDependencyFilter(Filter):
def __init__(self, lang: str = "en", min_dependency_num: int = 1, any_or_all: str = "all", *args, **kwargs):
Import
from data_juicer.ops.filter.text_entity_dependency_filter import TextEntityDependencyFilter
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| lang | str | No | Language for entity detection: "en" or "zh" (default: "en") |
| min_dependency_num | int | No | Minimum dependency edges per entity (default: 1) |
| any_or_all | str | No | Keep strategy: "any" or "all" (default: "all") |
Outputs
| Name | Type | Description |
|---|---|---|
| samples | Dict | Filtered samples with num_dependency_edges stat computed |
Usage Examples
YAML Configuration
process:
- text_entity_dependency_filter:
lang: en
min_dependency_num: 1
any_or_all: all
Python API
from data_juicer.ops.filter.text_entity_dependency_filter import TextEntityDependencyFilter
op = TextEntityDependencyFilter(lang="en", min_dependency_num=1)