Workflow:Volcengine Verl Supervised Fine Tuning
| Knowledge Sources | |
|---|---|
| Domains | LLMs, Fine_Tuning, Post_Training |
| Last Updated | 2026-02-07 18:00 GMT |
Overview
End-to-end process for supervised fine-tuning (SFT) of large language models using verl's FSDP-based SFT trainer, supporting both full-parameter and LoRA parameter-efficient fine-tuning.
Description
This workflow covers supervised fine-tuning of LLMs on task-specific datasets as a preparatory step before reinforcement learning. Using verl's dedicated SFT trainer with PyTorch FSDP for distributed training, the model learns from prompt-response pairs using standard cross-entropy loss. The workflow supports full-parameter fine-tuning, LoRA (Low-Rank Adaptation) for memory efficiency, sequence parallelism via DeepSpeed Ulysses, and multi-turn conversation data. SFT typically serves as the initial warm-up phase before RL post-training.
Usage
Execute this workflow when you have a dataset of high-quality prompt-response pairs and want to fine-tune a base or instruction-tuned model before applying reinforcement learning. This is the recommended first step in the post-training pipeline, especially when the base model needs to learn basic task formatting, domain-specific knowledge, or multi-turn conversation patterns before RL optimization.
Execution Steps
Step 1: Environment Setup
Install verl with the FSDP training backend. SFT does not require an inference engine (vLLM/SGLang) since there is no rollout generation. Configure distributed training parameters for torchrun (number of processes per node, number of nodes).
Key considerations:
- SFT uses torchrun for distributed execution (not Ray)
- Only the FSDP or Megatron backend is needed (no rollout engine required)
- GPU count determines the effective batch size via data parallelism
Step 2: Data Preparation
Prepare the training dataset in parquet format with prompt and response columns. For SFT, the data needs explicit prompt-response pairs rather than the RL format. The prompt_key and response_key configuration parameters map parquet columns to the trainer's expected input format.
Key considerations:
- Parquet files must contain identifiable prompt and response columns
- For multi-turn data, messages are stored as a list of role/content dictionaries
- The trainer supports configurable prompt_key and response_key for flexible column mapping
- Both train and test splits should be provided for validation during training
Step 3: Model and LoRA Configuration
Select the base model and configure training parameters. For parameter-efficient fine-tuning, set up LoRA with rank, alpha, and target modules. Full-parameter fine-tuning trains all model weights but requires more GPU memory.
Key considerations:
- LoRA typically targets all linear layers with rank 32 and alpha 16
- Learning rate for LoRA (1e-4) is typically higher than full fine-tuning (1e-5 to 5e-6)
- Sequence parallelism (Ulysses SP) can be enabled to handle long sequences across GPUs
- Liger kernel optimizations can be enabled for additional speedup
Step 4: Distributed Training
Run the SFT trainer with FSDP sharding across available GPUs. The trainer handles data loading, gradient accumulation, loss computation with cross-entropy, and distributed synchronization.
Key considerations:
- Micro batch size per GPU should be tuned to maximize GPU memory utilization
- Gradient accumulation enables larger effective batch sizes
- FSDP2 is recommended for better throughput and memory usage
- CPU offloading can be enabled to trade compute speed for memory
Step 5: Validation and Checkpoint Export
Evaluate the fine-tuned model on the test split and export the final checkpoint. For LoRA training, the adapter weights can be saved separately or merged into the base model weights for downstream use.
Key considerations:
- SFT checkpoints are in HuggingFace format and can be directly loaded for RL training
- LoRA adapters can be merged using verl's merge utilities
- The fine-tuned model serves as the starting point for GRPO or PPO training