Workflow:DataExpert io Data engineer handbook AB Experimentation Server
| Knowledge Sources | |
|---|---|
| Domains | Data_Engineering, Experimentation, AB_Testing |
| Last Updated | 2026-02-09 06:00 GMT |
Overview
End-to-end process for running a Flask-based A/B experimentation server that assigns users to experiment groups and tracks events using the Statsig platform.
Description
This workflow deploys a lightweight Flask web application that demonstrates A/B testing principles as part of Week 5 (KPIs and Experimentation) of the intermediate bootcamp. The server integrates with Statsig for experiment configuration and event tracking. It implements user assignment to experiment variants based on IP hashing, dynamic UI rendering based on experiment group (button colors, content filtering), and signup event logging for conversion analysis. The application serves as a practical demonstration of how experimentation infrastructure works in production systems.
Usage
Execute this workflow when you need to understand how A/B testing infrastructure works in practice, including user bucketing, variant assignment, event tracking, and experiment analysis. This is appropriate after completing the data processing modules (Weeks 1-4) and when ready to explore the analytics side of data engineering with KPI measurement and experiment design.
Execution Steps
Step 1: Install_Dependencies
Install the Python packages required to run the experimentation server. The requirements include Flask for the web framework, statsig for experiment management and event tracking, and gunicorn as a production-grade WSGI server.
Key considerations:
- requirements.txt specifies: flask, statsig, gunicorn, flask-cors
- A Statsig account and API key are required for experiment integration
- The STATSIG_API_KEY environment variable must be set before starting the server
Step 2: Configure_Statsig
Set up the Statsig platform integration by configuring the API key and defining the experiment. The server initializes the Statsig SDK on startup using the API key from the environment. Experiments (such as "button_color_v3") must be pre-configured in the Statsig dashboard with the desired variants and allocation rules.
What happens:
- statsig.initialize(API_KEY) connects to the Statsig backend
- Experiment "button_color_v3" is configured with variants for Button Color and Paragraph Text
- Variants include different colors (Blue, Green, Red, Orange) that affect UI rendering
- User assignment is deterministic based on hashed user ID
Step 3: Start_Flask_Server
Launch the Flask application in debug mode. The server exposes several endpoints: a root health check, a signup page with event tracking, a tasks listing with experiment-driven UI rendering, and CRUD endpoints for task management.
Available endpoints:
- GET / — Health check returning a greeting
- GET /signup — Signup page that logs a visited_signup event to Statsig
- GET /tasks — Task listing with experiment-driven styling and content filtering
- GET/POST/PUT/DELETE /tasks/<id> — Standard CRUD operations on in-memory task store
Step 4: User_Assignment_And_Bucketing
When users visit the application, they are assigned to experiment groups based on a deterministic hash of their IP address. The Statsig SDK returns the assigned variant values (button color, paragraph text) which drive the UI rendering. An optional random parameter enables generating random user IDs for testing multiple experiment groups.
What happens:
- User IP address is hashed to produce a stable user identifier
- StatsigUser object is created with the hashed user ID
- statsig.get_experiment returns variant values for the user's assigned group
- UI content is dynamically filtered: odd-ID tasks for blue/green groups, even-ID tasks for red/orange
- The ?random query parameter generates random user IDs for testing
Step 5: Track_Events_And_Analyze
Monitor experiment results by tracking user events through the Statsig event logging system. The signup endpoint logs a visited_signup event for each user visit, which can be analyzed in the Statsig dashboard for conversion rate comparison across experiment groups.
Key considerations:
- StatsigEvent objects capture user, event_name, and optional metadata
- statsig.log_event sends events asynchronously to the Statsig backend
- Analysis involves comparing conversion rates between experiment variants
- Experiment design follows standard A/B testing methodology with null/alternative hypotheses, leading/lagging metrics, and 50-50 allocation