Principle:DataExpert io Data engineer handbook Experiment User Assignment
Overview
Experiment User Assignment (also known as user bucketing) is the process by which an A/B testing system deterministically assigns each user to a specific experiment variant. This is the core mechanism that ensures users see consistent experiences and that experiment results are statistically valid.
Theory of User Assignment and Bucketing
In any A/B experiment, users must be divided into distinct groups (buckets) that each receive a different treatment. The assignment process must satisfy several properties:
- Determinism: The same user must always be assigned to the same variant for the duration of the experiment
- Uniformity: Users should be distributed evenly across variants (unless intentionally weighted)
- Independence: Assignment to one experiment should not influence assignment to another
- Low latency: Assignment must happen quickly, as it occurs in the critical path of request handling
Deterministic Assignment
Deterministic assignment ensures that a user who visits the application multiple times always sees the same variant. This is essential for:
- User experience consistency: A user should not see a blue button on one visit and a red button on the next
- Statistical validity: If users switch between variants, it contaminates the experiment data and makes causal inference unreliable
- Debugging and support: When investigating user-reported issues, deterministic assignment makes it possible to reproduce what the user saw
User Identity
To achieve deterministic assignment, each user needs a stable identifier. Common approaches include:
- IP address hashing: Using a hash of the user's IP address as a pseudo-anonymous identifier. This is simple but can group multiple users behind the same NAT or proxy into the same bucket.
- Random ID generation: Assigning a random identifier (e.g., via a query parameter or cookie) for cases where IP-based identification is insufficient or where anonymity is desired.
- Authenticated user ID: For logged-in users, the account ID provides the most stable and accurate identifier.
The choice of identity strategy involves trade-offs between accuracy, privacy, and implementation complexity.
Experiment Parameters
Once a user is assigned to a variant, the system retrieves the variant-specific parameters that define that variant's experience. These parameters can include:
- Visual elements: button colors, font sizes, layout configurations
- Text content: headlines, call-to-action text, paragraph content
- Behavioral settings: algorithm weights, feature toggles, thresholds
Parameters are defined in the experimentation platform's dashboard and fetched by the SDK at evaluation time.
When to Apply
This principle applies when:
- Serving different UI variants to different users in a web application
- Building any system that needs to split traffic between alternative implementations
- Implementing personalization logic that must be consistent per-user
Theoretical Basis
- Hash-Based Bucketing: The standard approach uses a hash function applied to the user identifier and experiment name. The hash output is mapped to a value between 0 and 1 (or 0 and 100), and this value determines which variant the user receives. Because hash functions are deterministic, the same input always produces the same output, guaranteeing consistent assignment.
- Experiment Groups (Control and Treatment): Every A/B experiment has at least two groups:
- The control group sees the existing (baseline) experience
- The treatment group (or groups) sees the new variant(s)
- Comparing outcomes between groups enables causal inference about the effect of the change