Implementation:Datajuicer Data juicer SentenceAugmentationMapper

Knowledge Sources	Datajuicer_Data_juicer
Domains	Data_Processing, Mapping
Last Updated	2026-02-14 16:00 GMT

Overview

Concrete tool for augmenting sentences using language models provided by Data-Juicer.

Description

SentenceAugmentationMapper is a mapper operator that augments individual sentences by generating enhanced or paraphrased versions using a Hugging Face language model. It loads a HF model (default: Qwen/Qwen2-7B-Instruct) with configurable generation parameters (temperature, top_p, num_beams, max_new_tokens), applies a system prompt and task instruction to each input sentence, and stores the model-generated augmentation in text_key_second. It supports both Vicuna-style and chat-template-based prompt formatting. Requires CUDA acceleration and both text keys to be set.

Usage

Use when you need to create diverse training pairs through data augmentation, particularly for sentence-level tasks like paraphrase generation and contrastive learning.

Code Reference

Source Location

• Repository: Datajuicer_Data_juicer
• File: data_juicer/ops/mapper/sentence_augmentation_mapper.py

Signature

@OPERATORS.register_module("sentence_augmentation_mapper")
class SentenceAugmentationMapper(Mapper):
    def __init__(
        self,
        hf_model: str = "Qwen/Qwen2-7B-Instruct",
        system_prompt: str = None,
        task_sentence: str = None,
        max_new_tokens=256,
        temperature=0.2,
        top_p=None,
        num_beams=1,
        text_key=None,
        text_key_second=None,
        *args,
        **kwargs,
    ):

Import

from data_juicer.ops.mapper.sentence_augmentation_mapper import SentenceAugmentationMapper

I/O Contract

Inputs

Name	Type	Required	Description
hf_model	str	No	HuggingFace model ID (default: Qwen/Qwen2-7B-Instruct)
system_prompt	str	No	System prompt for the model (default: built-in assistant prompt)
task_sentence	str	Yes	The instruction for the current augmentation task
max_new_tokens	int	No	Maximum number of new tokens generated (default: 256)
temperature	float	No	Controls randomness of generation (default: 0.2)
top_p	float	No	Nucleus sampling probability threshold (default: None)
num_beams	int	No	Beam search size (default: 1)
text_key	str	No	Key name for the input sentence (default: text)
text_key_second	str	Yes	Key name for storing the augmented sentence

Outputs

Name	Type	Description
sample[text_key_second]	str	Augmented version of the input sentence

Usage Examples

process:
  - sentence_augmentation_mapper:
      hf_model: 'Qwen/Qwen2-7B-Instruct'
      task_sentence: 'Paraphrase the given sentence while preserving its meaning.'
      text_key: 'text'
      text_key_second: 'augmented_text'
      temperature: 0.2
      max_new_tokens: 256

Related Pages

• Environment:Datajuicer_Data_juicer_Python_Runtime_Environment