AB Testing

Downloadable Code Files

Getting Started with Code Files

Getting Started

Introduction

Experiment Goals

Key Members

Product KPI

The North Star Metric

Driver Metric

Guardrail Metrics

Secondary Metrics

Segmentation Metrics

Launch Criteria

Experiment Data

Amazon AB Testing Case - Question

Amazon AB Testing Case - Solution

Code Walkthrough

Understand the Business Problem

State the Experiment Hypothesis

Hypothesis Statement

Significance Level

Statistical Power

Minimum Detectable Effect (MDE)

Design the Experiment

Randomization Unit

Sample Size Calculation

Experiment Duration

Run the Experiment

Experiment Platform

Peeking at P-Values

Assess Validity Threats

Stable Unit Treatment Value Assumption

Survivorship Bias

Sample Ratio Mismatch

Primacy Effect

Novelty Effect

Holiday Effect

Other Effects to Consider

AA Test

Conduct Statistical Inference

Chi-Squared Test of Independence

Z-Test for Proportions

T-Test for Means

Decide Whether to Launch

Interpret Test Results (Relative vs Absolute Difference)

Launch Decisions

Decision Rule

Decision Rule with Confidence Interval

Holdout Set

[Google] An experiment is statistically significant, but the sample is low. Should you change?

[Amazon] An AB test on "frequently bought together" showed improve in test, but not after launch. What happened?

[Amazon] New UI and algorithm are tested on product search. How would you design an experiment?

[Facebook] An experiment on News Feed showed increase in CTR, but the p-value is 0.06. Should you launch the change?

[Instacart] What are important statistical parameters in AB testing?

Interview Cases

[Amazon] Testing Recommender Systems

[Uber] Improving Driver Enrollment Rate

Mock Interviews

Learn to think like a product data scientist at large-scale online platforms such as Google, Amazon, and TikTok. Taught by a former data scientist at Google, the course gives you a hands-on experience of designing and solving a real-world AB testing case using Python. It contains over 70+ video lessons (e.g. conceptual deep dives and mock interview videos), coding case and practice cases - designed to help you grasp the foundation of AB Testing and become interview ready for AB testing interviews. It's a must-have for data scientists and data analytics professionals!

Understand Business Problem

Amazon AB Testing: Question

Amazon AB Testing: Solution

Hypothesis Statement: Question 1

Chi-Squared Test of Independence: Question

Hypothesis Statement: Solution 1

Hypothesis Statement: Question 2

Hypothesis Statement: Solution 2

Significance Level: Question

Significance Level: Solution

Statistical Power: Question 1

Statistical Power: Solution 1

Statistical Power: Question 2

Statistical Power: Solution 2

Minimum Detectable Effect: Question

Minimum Detectable Effect: Solution

Sample Size Calculation: Question

Sample Size Calculation: Solution

Experiment Duration: Question

Experiment Duration: Solution

Chi-Squared Test of Independence: Solution

Z-Test for Proportions: Question

Z-Test for Proportions: Solution

T-Test for Means: Question

T-Test for Means: Solution

Launch Decision

Interpret Test: Relative Difference

Confidence Interval

Practice Cases

Peeking at the P-Value

Failed to Generalize

Multiple Testing

No Statistical Signifiance

Statistical Parameters

Amazon Testing Recommender System

Improve Driver Enrollments on Uber