Principle:SeldonIO Seldon core Experiment Execution
| Field | Value |
|---|---|
| Overview | Activating an experiment to begin traffic splitting or mirroring between model candidates. |
| Domains | MLOps, Experimentation |
| Related Implementation | SeldonIO_Seldon_core_Seldon_Experiment_Start |
| Last Updated | 2026-02-13 00:00 GMT |
Description
Starting an experiment activates the traffic routing rules defined in the Experiment CRD. The Seldon scheduler intercepts requests to the default model endpoint and routes them to candidates based on their configured weights. The experiment runs continuously until explicitly stopped or updated.
When an experiment is activated:
- The scheduler registers the experiment in its routing table
- Requests to the default model endpoint are intercepted by the experiment router
- Each request is probabilistically assigned to a candidate based on the weight distribution
- The assigned candidate processes the request and returns its response
- The response includes metadata (via the
x-seldon-routeheader) indicating which candidate served it - In mirror mode, the primary candidate's response is returned while a copy of the request is asynchronously sent to the mirror target
The experiment remains active until one of the following occurs:
- The experiment is explicitly stopped via
seldon experiment stop - The experiment is deleted via
kubectl delete - The experiment CRD is updated with new parameters (triggering a reconfiguration)
Theoretical Basis
Experiment activation creates a routing layer in the request path. The scheduler's routing table is updated to probabilistically forward requests to different candidates. This is implemented at the infrastructure level (not application level) to avoid model-specific changes.
Key theoretical properties of experiment execution:
- Transparent interception: The routing layer is inserted between the client and the model serving layer. Clients are unaware that an experiment is active; they continue sending requests to the same endpoint.
- Probabilistic routing: Each request is independently routed to a candidate based on the weight distribution. Over a large number of requests, the observed traffic distribution converges to the configured weights.
- Stateless routing: By default, each request is routed independently. There is no affinity between requests from the same client (unless sticky sessions are explicitly configured via the
x-seldon-routeheader). - Infrastructure-level implementation: The experiment routing is handled by the scheduler, not by the models themselves. This means models do not need to be modified or aware that they are part of an experiment.
- Idempotent activation: Starting an experiment that is already active is a no-op (or updates it if the configuration changed). This supports declarative, convergent operations.
Usage
This principle applies after defining an Experiment CRD and ensuring all candidates are deployed. The typical workflow is:
- Verify all candidate models are loaded and ready
- Apply the Experiment CRD to start traffic splitting
- Monitor traffic distribution using inference with headers
- Analyze results and determine the winning candidate
- Stop the experiment and promote the winner
Experiment execution requires:
- All candidate models/pipelines referenced in the experiment must be in a ready state
- The Seldon scheduler must be running and accessible
- The experiment manifest must be valid (correct API version, kind, and spec fields)
If a referenced candidate is not available when the experiment starts, the scheduler may reject the experiment or route traffic only to available candidates, depending on configuration.
Related Pages
- SeldonIO_Seldon_core_Seldon_Experiment_Start — implements this principle — Concrete CLI tool for starting A/B test and traffic mirroring experiments in Seldon Core 2.
- SeldonIO_Seldon_core_Experiment_Configuration — prerequisite principle — Declarative specification of traffic routing rules for A/B tests and traffic mirroring experiments.
- SeldonIO_Seldon_core_Experiment_Traffic_Analysis — next principle — Monitoring which candidate model serves each request during an experiment.
- SeldonIO_Seldon_core_Experiment_Lifecycle_Management — related principle — Operational procedures for updating and concluding experiments.