Implementation:OpenHands OpenHands SaaSExperimentManager
| Knowledge Sources | |
|---|---|
| Domains | Experimentation, SaaS_Infrastructure |
| Last Updated | 2026-02-11 21:00 GMT |
Overview
Concrete tool for running A/B test experiment variants against different agent configurations, provided by the OpenHands enterprise experimentation layer.
Description
The SaaSExperimentManager class orchestrates variant-based experimentation for agent configurations in the SaaS platform. It enables controlled comparison of different agent settings, conversation parameters, and configuration options by routing traffic to distinct experiment variants and collecting performance metrics.
The primary entry point run_agent_variant_tests__v1 executes a full suite of agent variant tests, coordinating the allocation of users or sessions to experiment buckets and collecting outcome data. The run_conversation_variant_test method focuses on conversation-level experiments, comparing how different conversation parameters (such as system prompts, context window sizes, or turn limits) affect agent performance. The run_config_variant_test method operates at the configuration level, testing variations in agent runtime settings such as model selection, temperature, or tool availability.
Each method follows a consistent pattern: resolve the active experiment definition, assign the incoming request to a variant, apply the variant-specific configuration overrides, execute the agent interaction, and record the outcome metrics for later analysis.
Usage
Use SaaSExperimentManager when rolling out changes to agent configurations that require controlled evaluation before full deployment. Instantiate the manager with the current experiment definitions and call the appropriate variant test method based on the scope of the experiment (agent-level, conversation-level, or config-level). Results are collected asynchronously and can be queried through the experiment analytics pipeline.
Code Reference
Source Location
- Repository: OpenHands
- File: enterprise/experiments/experiment_manager.py
- Lines: 1-97
Signature
class SaaSExperimentManager:
def run_agent_variant_tests__v1(
self,
experiment_id: str,
session_context: dict,
) -> dict:
...
def run_conversation_variant_test(
self,
experiment_id: str,
conversation_params: dict,
) -> dict:
...
def run_config_variant_test(
self,
experiment_id: str,
config_overrides: dict,
) -> dict:
...
Import
from enterprise.experiments.experiment_manager import SaaSExperimentManager
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| experiment_id | str |
Yes | Unique identifier for the experiment definition to execute |
| session_context | dict |
Yes | Contextual data about the current session used for variant assignment (for run_agent_variant_tests__v1) |
| conversation_params | dict |
Yes | Parameters governing conversation behavior to be varied across experiment arms (for run_conversation_variant_test) |
| config_overrides | dict |
Yes | Configuration key-value pairs to apply as variant-specific overrides (for run_config_variant_test) |
Outputs
| Name | Type | Description |
|---|---|---|
| result | dict |
Experiment outcome containing the assigned variant identifier, applied configuration, and collected performance metrics |
Usage Examples
from enterprise.experiments.experiment_manager import SaaSExperimentManager
# Initialize the experiment manager
experiment_mgr = SaaSExperimentManager()
# Run an agent-level A/B test
result = experiment_mgr.run_agent_variant_tests__v1(
experiment_id="exp-agent-model-comparison-2026",
session_context={"user_id": "u-123", "org_id": "org-456"},
)
# Run a conversation-level variant test
conv_result = experiment_mgr.run_conversation_variant_test(
experiment_id="exp-system-prompt-v2",
conversation_params={"system_prompt": "You are a helpful assistant."},
)
# Run a config-level variant test
config_result = experiment_mgr.run_config_variant_test(
experiment_id="exp-temperature-sweep",
config_overrides={"temperature": 0.7, "model": "gpt-4"},
)