Workflow:Volcengine Verl Data Preprocessing For RL
| Knowledge Sources | |
|---|---|
| Domains | Data_Engineering, LLMs, Reinforcement_Learning |
| Last Updated | 2026-02-07 18:00 GMT |
Overview
End-to-end process for converting raw datasets from HuggingFace or local sources into verl's standardized parquet format for reinforcement learning training, including reward function configuration and data schema validation.
Description
This workflow transforms raw datasets (math problems, coding tasks, RLHF preference data, multimodal inputs) into the standardized parquet format required by verl's RL trainers. Each record in the output contains a structured chat-format prompt, data source identifier, ability classification, reward model configuration (rule-based with ground truth or model-based), and extra metadata. The preprocessing pipeline supports single-turn, multi-turn with tool use, and multimodal (vision-language) data formats.
Usage
Execute this workflow before any RL training run (GRPO, PPO, REINFORCE++, etc.) when you have a raw dataset that needs to be converted to verl's parquet format. This is required for every new dataset or when modifying the prompt template, reward configuration, or data splits.
Execution Steps
Step 1: Dataset Acquisition
Load the raw dataset from HuggingFace Hub, a local file path, or a custom data source. Identify the relevant fields (questions, answers, solutions, preferences) and determine the train/test split strategy.
Key considerations:
- HuggingFace datasets are loaded using the datasets library with specified splits
- Local datasets can be loaded from JSON, CSV, or other formats
- Some datasets require authentication or special access (e.g., gated models)
- Determine the data split ratio (e.g., full dataset for train, separate test set)
Step 2: Prompt Template Design
Design the chat-format prompt template that converts raw data fields into the structured message format expected by the target model. This includes crafting system prompts (for multi-turn), user messages with task-specific instructions, and any special formatting tokens.
Key considerations:
- Prompts are stored as a list of message dictionaries with role and content fields
- For math tasks, append instructions like "Let's think step by step and output the final answer after ####"
- For multi-turn with tools, include a system prompt describing available tools
- The prompt template should match the model's expected chat format
Step 3: Answer Extraction and Ground Truth Parsing
Parse raw answers to extract structured ground truth values used by the reward function. For math datasets, this means extracting numerical answers using regex patterns. For coding tasks, this may involve extracting test cases or expected outputs.
Key considerations:
- Math answers typically use patterns like "#### (number)" to mark the final answer
- Ground truth values must be clean and normalized for reliable reward computation
- Handle edge cases (negative numbers, decimals, commas in numbers)
- Store both the extracted answer and the full solution for debugging
Step 4: Reward Configuration
Configure the reward model specification for each data record. Choose between rule-based rewards (with ground truth for exact match) or model-based rewards (using a learned reward model). For multi-turn data, configure tool reward functions and interaction kwargs.
Key considerations:
- Rule-based rewards set style to "rule" and include ground_truth in the reward_model field
- Model-based rewards set style to "model" (reward signal comes from a trained RM)
- Multi-turn tool rewards include tools_kwargs specifying tool initialization and reward extraction
- Custom reward functions can be registered for domain-specific evaluation
Step 5: Schema Assembly and Parquet Export
Assemble the final data schema with all required columns (data_source, prompt, ability, reward_model, extra_info) and export to parquet format. Validate the schema against verl's expected format and verify data integrity.
Key considerations:
- Required columns: data_source (str), prompt (list of dicts), ability (str), reward_model (dict), extra_info (dict)
- Extra_info preserves original data fields for debugging and evaluation
- Parquet files should be written to a path accessible by the training cluster
- HDFS copying is supported for distributed storage environments
- Generate both train.parquet and test.parquet for training and evaluation