Workflow:Volcengine Verl Multi Turn Tool Use Training
| Knowledge Sources | |
|---|---|
| Domains | LLMs, Reinforcement_Learning, Agentic_RL, Tool_Use |
| Last Updated | 2026-02-07 18:00 GMT |
Overview
End-to-end process for training LLMs to use external tools through multi-turn reinforcement learning, where the model learns to iteratively invoke tools (code execution, search, calculators) and incorporate their results across conversation turns.
Description
This workflow enables training LLMs for agentic behavior by combining multi-turn rollout generation with tool-calling capabilities. Using SGLang as the rollout engine, the model generates responses that may include tool invocations. The tool server executes these calls and returns results, which are appended to the conversation for the next turn. After the multi-turn interaction completes, the entire trajectory is scored and used for GRPO-based policy optimization. This supports tools like Python code sandboxes (Sandbox Fusion), web search APIs, MCP servers, and custom interaction handlers.
Usage
Execute this workflow when you want to train an LLM to effectively use external tools during reasoning, such as a calculator for math problems, a code execution sandbox for programming tasks, or a search engine for knowledge-intensive questions. This is particularly useful for building agentic AI systems that learn to decompose problems and leverage tools iteratively.
Execution Steps
Step 1: Environment Setup with SGLang
Install verl with the SGLang inference backend, which provides native multi-turn conversation support. Configure tool servers (Sandbox Fusion, MCP servers, or custom tool handlers) that the model will interact with during rollout.
Key considerations:
- SGLang is required for multi-turn rollout (vLLM does not support multi-turn natively in this context)
- Tool servers must be running and accessible before training starts
- MCP server configuration requires a JSON config file specifying server endpoints
- Sandbox Fusion requires a running sandbox service for code execution
Step 2: Multi-Turn Data Preparation
Preprocess datasets into multi-turn format with tool configuration metadata. Each data record includes a system prompt instructing the model to use tools, the user query, and tool configuration specifying which tools are available and how rewards are computed based on tool interactions.
Key considerations:
- System prompts must describe available tools and their usage format
- Tool kwargs in extra_info specify tool initialization parameters (e.g., ground truth for verification)
- The tool_config YAML file maps tool names to their implementation classes and reward functions
- Interaction kwargs enable custom conversation handlers for complex multi-turn scenarios
Step 3: Tool Configuration
Define the tool configuration YAML that maps tool names to their Python implementations, specifying how tools are created, invoked, and how rewards are extracted from tool execution results. Each tool entry includes the tool class path, creation kwargs, and reward extraction logic.
Key considerations:
- Tool configs reference Python classes that implement the tool interface
- Each tool specifies create_kwargs for initialization and reward extraction methods
- Multiple tools can be configured for a single training run
- MCP tools use a separate JSON configuration for server endpoints and authentication
Step 4: Multi-Turn Rollout Generation
Generate multi-turn conversations where the model produces responses, tool calls are detected and executed, and results are fed back for subsequent turns. The SGLang engine manages the conversation state, handling tool call parsing, execution, and response formatting across multiple turns.
Key considerations:
- The number of turns is bounded by the maximum response length and turn limits
- Tool calls are detected using the model's native tool-calling format (e.g., function calling syntax)
- Async rollout mode with oversampling improves throughput for multi-turn generation
- Memory profiling is recommended due to the variable-length nature of multi-turn conversations
Step 5: Trajectory Reward Computation
Score the complete multi-turn trajectory based on the final outcome. For math problems, this means checking whether the model arrived at the correct answer (potentially through multiple tool-assisted reasoning steps). Reward shaping can optionally assign intermediate rewards for tool usage quality.
Key considerations:
- Trajectory-level rewards evaluate the complete multi-turn conversation
- Tool-call shaping rewards can encourage appropriate tool usage patterns
- The reward function receives the full conversation history and tool execution results
- Custom reward functions can combine outcome accuracy with process-quality signals
Step 6: GRPO Policy Update
Compute group-relative advantages across the sampled trajectories and update the actor policy using the GRPO algorithm. The policy learns to produce better tool-calling strategies and reasoning chains through the reinforcement signal from trajectory outcomes.
Key considerations:
- GRPO with multi-turn uses the same advantage estimation as single-turn
- Token-level log probabilities span the entire multi-turn conversation
- KL regularization prevents the policy from collapsing to degenerate tool-calling patterns
- Curriculum learning can progressively increase problem difficulty during training
Step 7: Evaluation and Iteration
Evaluate the trained model on held-out problems requiring tool use. Monitor metrics including answer accuracy, tool usage frequency, average number of turns, and reward trends to assess whether the model is learning effective tool-calling strategies.
Key considerations:
- Multi-turn evaluation requires running the full tool interaction loop
- Track tool usage statistics alongside accuracy metrics
- Compare against single-turn baselines to quantify the benefit of tool use
- Experiment tracking via MLflow or WandB captures multi-turn specific metrics