Principle:SeldonIO Seldon core Experiment Configuration
| Field | Value |
|---|---|
| Overview | Declarative specification of traffic routing rules for A/B tests and traffic mirroring experiments. |
| Domains | MLOps, Experimentation |
| Related Implementation | SeldonIO_Seldon_core_Seldon_Experiment_CRD |
| Last Updated | 2026-02-13 00:00 GMT |
Description
Seldon Core 2 uses an Experiment CRD (kind: Experiment) to declare traffic splitting between candidates. Each candidate has a weight determining its traffic share. A default candidate can reuse an existing model endpoint, ensuring that inference requests sent to the default model name are transparently routed according to experiment weights. Mirror mode sends shadow traffic to a secondary model without affecting responses returned to the caller.
The Experiment resource supports two primary modes:
- A/B Testing (traffic splitting): Requests to the default endpoint are probabilistically routed to one of the listed candidates based on their relative weights. The response comes from whichever candidate was selected for that request.
- Traffic Mirroring: All requests are served by the primary candidate, but a configurable percentage of requests are also duplicated and sent to a mirror target. The mirror's responses are discarded (not returned to the caller) but can be logged for offline analysis.
Experiments can also operate on pipelines (not just individual models) by setting spec.resourceType: pipeline.
Theoretical Basis
Traffic splitting distributes requests probabilistically across candidates based on weights. The weight ratio determines the traffic percentage each candidate receives:
traffic_percentage = candidate_weight / sum(all_candidate_weights)
For example, if candidate A has weight 50 and candidate B has weight 50, each receives 50% of traffic. If candidate A has weight 90 and candidate B has weight 10, the split is 90/10.
Mirroring duplicates requests to a shadow target for offline comparison without impacting production responses. The mirror receives a copy of the request at the specified percentage (0-100%), but its response is never returned to the caller. This enables:
- Safe evaluation of new models against live production traffic
- Performance benchmarking without risk to production quality
- Data collection for offline A/B analysis
Key theoretical properties:
- Declarative routing: The experiment defines what the traffic split should be, not how to implement it. The scheduler handles the routing mechanics.
- Weight normalization: Weights are relative, not absolute percentages. The scheduler normalizes them to produce the correct distribution.
- Endpoint transparency: Callers send requests to the same endpoint (the default model name) regardless of whether an experiment is active. The experiment is invisible to the client.
Usage
This principle applies when defining how traffic should be distributed between model variants for comparison. Use it when:
- Designing an A/B test between two or more model candidates
- Setting up a canary deployment with weighted traffic shifting
- Configuring traffic mirroring for shadow evaluation of a new model
- Running pipeline-level experiments (comparing different inference pipelines)
The configuration is expressed as a YAML manifest containing:
spec.default— the model name whose endpoint will be shared by the experimentspec.candidates— a list of candidate models/pipelines with their weightsspec.mirror(optional) — a mirror target with a duplication percentagespec.resourceType(optional) — set topipelinefor pipeline experiments
Related Pages
- SeldonIO_Seldon_core_Seldon_Experiment_CRD — implements this principle — Concrete pattern for declaring Seldon Core 2 Experiment resources as Kubernetes YAML manifests.
- SeldonIO_Seldon_core_Candidate_Model_Deployment — prerequisite principle — Deploying multiple model variants as candidates for A/B testing or traffic mirroring experiments.
- SeldonIO_Seldon_core_Experiment_Execution — next principle — Activating an experiment to begin traffic splitting or mirroring between model candidates.