Principle:DataExpert io Data engineer handbook Event Tracking
Overview
Event Tracking in A/B experiments is the practice of logging user actions that are associated with specific experiment variants. Without event tracking, experiments produce no measurable outcomes, making it impossible to determine which variant performs better.
Theory of Event Tracking in A/B Experiments
Event tracking is the measurement layer of an experimentation system. While user assignment determines what a user sees, event tracking records what the user does in response. This data is the foundation for all experiment analysis and decision-making.
The core idea is straightforward: every meaningful user action is recorded as a structured event, tagged with the user's identity (and implicitly, their experiment variant assignment). These events are then aggregated and analyzed to compare variant performance.
Logging User Actions Associated with Experiment Variants
Each tracked event captures:
- Who: The user identifier (the same ID used for experiment bucketing)
- What: The event name describing the action (e.g.,
visited_signup,clicked_button,completed_purchase) - When: The timestamp of the event (typically added automatically by the SDK)
- Context: The experiment variant the user was assigned to (linked automatically by the experimentation platform via the user ID)
Because the same user ID is used for both experiment assignment and event logging, the experimentation platform can automatically associate events with the correct variant without explicit variant tagging in each event.
Enables Post-Hoc Analysis of Experiment Impact
Events are not analyzed in real time during the experiment (in most cases). Instead, they are collected over the experiment's duration and analyzed after the fact (post-hoc) to determine:
- Whether the treatment variant produced a statistically significant difference compared to the control
- The magnitude of the effect (e.g., 5% increase in signups)
- Whether the result is practically significant (large enough to matter for the business)
Event Types
Common event types in A/B experiments include:
- Page visits: Tracking which pages users view (e.g.,
visited_signupindicates the user reached the signup page) - Conversions: Key actions that represent experiment goals (e.g., completing a purchase, submitting a form)
- Engagement actions: Interactions that indicate user interest (e.g., clicks, scrolls, time on page)
- Error events: Tracking failures or errors that may differ between variants
When to Apply
This principle applies when:
- Measuring user behavior within experiment variants to determine which variant is more effective
- Building any system where decisions must be data-driven rather than based on opinion
- Implementing conversion funnels that need to be compared across experiment groups
Theoretical Basis
- Experiment Metrics: Every experiment defines one or more metrics — quantitative measures derived from tracked events. The primary metric (e.g., conversion rate) is what the experiment is designed to move. Secondary metrics help detect unintended side effects.
- Conversion Tracking: A conversion is a user action that represents success for the experiment's goal. Conversion rate (conversions divided by total users in a variant) is the most common metric for comparing variants.
- Statistical Significance: To determine whether an observed difference between variants is real (not due to random chance), statistical tests are applied to the event data. This requires a sufficient volume of events (sample size) and a pre-determined significance threshold (typically p < 0.05).