Principle:SeldonIO Seldon core Experiment Execution

Field	Value
Overview	Activating an experiment to begin traffic splitting or mirroring between model candidates.
Domains	MLOps, Experimentation
Related Implementation	SeldonIO_Seldon_core_Seldon_Experiment_Start
Last Updated	2026-02-13 00:00 GMT

Description

Starting an experiment activates the traffic routing rules defined in the Experiment CRD. The Seldon scheduler intercepts requests to the default model endpoint and routes them to candidates based on their configured weights. The experiment runs continuously until explicitly stopped or updated.

When an experiment is activated:

• The scheduler registers the experiment in its routing table
• Requests to the default model endpoint are intercepted by the experiment router
• Each request is probabilistically assigned to a candidate based on the weight distribution
• The assigned candidate processes the request and returns its response
• The response includes metadata (via the x-seldon-route header) indicating which candidate served it
• In mirror mode, the primary candidate's response is returned while a copy of the request is asynchronously sent to the mirror target

The experiment remains active until one of the following occurs:

1. The experiment is explicitly stopped via seldon experiment stop
2. The experiment is deleted via kubectl delete
3. The experiment CRD is updated with new parameters (triggering a reconfiguration)

Theoretical Basis

Experiment activation creates a routing layer in the request path. The scheduler's routing table is updated to probabilistically forward requests to different candidates. This is implemented at the infrastructure level (not application level) to avoid model-specific changes.

Key theoretical properties of experiment execution:

• Transparent interception: The routing layer is inserted between the client and the model serving layer. Clients are unaware that an experiment is active; they continue sending requests to the same endpoint.
• Probabilistic routing: Each request is independently routed to a candidate based on the weight distribution. Over a large number of requests, the observed traffic distribution converges to the configured weights.
• Stateless routing: By default, each request is routed independently. There is no affinity between requests from the same client (unless sticky sessions are explicitly configured via the x-seldon-route header).
• Infrastructure-level implementation: The experiment routing is handled by the scheduler, not by the models themselves. This means models do not need to be modified or aware that they are part of an experiment.
• Idempotent activation: Starting an experiment that is already active is a no-op (or updates it if the configuration changed). This supports declarative, convergent operations.

Usage

This principle applies after defining an Experiment CRD and ensuring all candidates are deployed. The typical workflow is:

1. Verify all candidate models are loaded and ready
2. Apply the Experiment CRD to start traffic splitting
3. Monitor traffic distribution using inference with headers
4. Analyze results and determine the winning candidate
5. Stop the experiment and promote the winner

Experiment execution requires:

• All candidate models/pipelines referenced in the experiment must be in a ready state
• The Seldon scheduler must be running and accessible
• The experiment manifest must be valid (correct API version, kind, and spec fields)

If a referenced candidate is not available when the experiment starts, the scheduler may reject the experiment or route traffic only to available candidates, depending on configuration.

Related Pages

• SeldonIO_Seldon_core_Seldon_Experiment_Start — implements this principle — Concrete CLI tool for starting A/B test and traffic mirroring experiments in Seldon Core 2.
• SeldonIO_Seldon_core_Experiment_Configuration — prerequisite principle — Declarative specification of traffic routing rules for A/B tests and traffic mirroring experiments.
• SeldonIO_Seldon_core_Experiment_Traffic_Analysis — next principle — Monitoring which candidate model serves each request during an experiment.
• SeldonIO_Seldon_core_Experiment_Lifecycle_Management — related principle — Operational procedures for updating and concluding experiments.

Implementation:SeldonIO_Seldon_core_Seldon_Experiment_Start