Implementation:Datajuicer Data juicer FixUnicodeMapper

Knowledge Sources	Datajuicer_Data_juicer
Domains	Data_Processing, Mapping
Last Updated	2026-02-14 16:00 GMT

Overview

Concrete tool for fixing Unicode errors and normalizing text provided by Data-Juicer.

Description

FixUnicodeMapper is a mapper operator that corrects common Unicode errors and normalizes text to a specified Unicode normalization form. It uses the ftfy (fixes text for you) library to repair mojibake and other Unicode encoding issues in each text sample. The default normalization form is NFC, but it can be set to NFKC, NFD, or NFKD. Operates in batched mode, processing all texts in a batch with a single pass. Raises a ValueError if an unsupported normalization form is provided.

Usage

Use when processing web-scraped or multi-source datasets that contain encoding corruption, to ensure consistent Unicode representation across all text samples.

Code Reference

Source Location

• Repository: Datajuicer_Data_juicer
• File: data_juicer/ops/mapper/fix_unicode_mapper.py

Signature

@OPERATORS.register_module("fix_unicode_mapper")
class FixUnicodeMapper(Mapper):
    def __init__(self, normalization: str = None, *args, **kwargs):

Import

from data_juicer.ops.mapper.fix_unicode_mapper import FixUnicodeMapper

I/O Contract

Inputs

Name	Type	Required	Description
normalization	str	No	Unicode normalization mode, one of NFC, NFKC, NFD, NFKD; defaults to NFC

Outputs

Name	Type	Description
samples	Dict	Transformed samples with fixed Unicode text

Usage Examples

process:
  - fix_unicode_mapper:
      normalization: "NFC"

Related Pages

• Environment:Datajuicer_Data_juicer_Python_Runtime_Environment