Implementation:OpenRLHF OpenRLHF GEM Multiturn AgentExecutor

Knowledge Sources	OpenRLHF
Domains	Reinforcement_Learning, Agent_Execution, Multi_Turn
Last Updated	2026-02-07 10:40 GMT

Overview

Concrete tool for executing multi-turn GEM (Game Environment for Models) agent rollouts with step-level rewards.

Description

The AgentInstance class implements a multi-turn agent that interacts with GEM game environments (specifically GuessTheNumber-v0). At each turn, the agent receives an observation, generates an action via the policy model, and receives reward feedback from the environment. It extends AgentInstanceBase and is wrapped by AgentExecutor (extending MultiTurnAgentExecutor) for integration with OpenRLHF's PPO training pipeline. The class uses Qwen3-style chat templates for formatting observations.

Usage

Use this executor when training a language model with multi-turn reinforcement learning on GEM game environments. The agent interacts over multiple steps, accumulating rewards per turn, which enables training with step-level reward signals rather than episode-level rewards only.

Code Reference

Source Location

Repository: OpenRLHF
File: examples/python/agent_func_gem_multiturn.py
Lines: 1-144

Signature

class AgentInstance(AgentInstanceBase):
    async def __init__(self, *args, **kwargs): ...
    async def reset(self, states: dict, **kwargs) -> dict: ...
    async def step(self, states: dict, **kwargs) -> Dict[str, Any]: ...

class AgentExecutor(MultiTurnAgentExecutor):
    def __init__(self): ...

Import

from examples.python.agent_func_gem_multiturn import AgentExecutor, AgentInstance

I/O Contract

Inputs (AgentInstance.step)

Name	Type	Required	Description
states["observation_text"]	str	Yes	Current observation from the environment
states["action_text"]	str	Yes	Generated action text from the policy model
states["label"]	str	Yes	Ground truth label for the episode
states["sampling_params"]	SamplingParams	No	vLLM sampling parameters for next step

Outputs (AgentInstance.step)

Name	Type	Description
rewards	torch.Tensor	Reward value for advantage calculation
scores	torch.Tensor	Reward value for dynamic filtering
environment_feedback	str	Formatted environment observation for next turn
done	bool	Whether the episode has terminated
sampling_params	SamplingParams	Parameters for vLLM sampling in next step
extra_logs	dict	Contains dummy_scores and turn_count tensors

Usage Examples

Setting Up GEM Multi-Turn Agent

from examples.python.agent_func_gem_multiturn import AgentExecutor

# The AgentExecutor wraps AgentInstance for use with OpenRLHF PPO training
executor = AgentExecutor()

# Used internally by OpenRLHF's experience maker during PPO rollouts.
# The executor manages multi-turn interactions with the GEM environment:
# 1. reset() initializes the GuessTheNumber game
# 2. step() processes each agent action and returns environment feedback
# 3. Rewards are accumulated per turn for step-level RL training

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment