Introduction

Downloadable Code Files

Getting Started with Code Files

Understand Business Problem

Experiment Goals

Key Members

Product KPI

The North Star Metric

Driver Metric

Guardrail Metrics

Secondary Metrics

Segmentation Metrics

Launch Criteria

Experiment Data

Amazon AB Testing: Question

Amazon AB Testing: Solution

Code Walkthrough

State the Experiment Hypothesis

Hypothesis Statement

Hypothesis Statement: Question 1

Chi-Squared Test of Independence: Question

Hypothesis Statement: Solution 1

Hypothesis Statement: Question 2

Hypothesis Statement: Solution 2

Significance Level

Significance Level: Question

Significance Level: Solution

Statistical Power

Statistical Power: Question 1

Statistical Power: Solution 1

Statistical Power: Question 2

Statistical Power: Solution 2

Minimum Detectable Effect (MDE)

Minimum Detectable Effect: Question

Minimum Detectable Effect: Solution

Design the Experiment

Randomization Unit

Sample Size Calculation

Sample Size Calculation: Question

Sample Size Calculation: Solution

Experiment Duration

Experiment Duration: Question

Experiment Duration: Solution

Run the Experiment

Experiment Platform

Peeking at P-Values

Assess Validity Threats

Stable Unit Treatment Value Assumption

Survivorship Bias

Sample Ratio Mismatch

Primacy Effect

Novelty Effect

Holiday Effect

Other Effects to Consider

AA Test

Conduct Statistical Inference

Chi-Squared Test of Independence

Chi-Squared Test of Independence: Solution

Z-Test for Proportions

Z-Test for Proportions: Question

Z-Test for Proportions: Solution

T-Test for Means

T-Test for Means: Question

T-Test for Means: Solution

Launch Decision

Decide Whether to Launch

Interpret Test: Relative Difference

Launch Decisions

Decision Rule

Confidence Interval

Holdout Set

Practice Cases

Peeking at the P-Value

Failed to Generalize

Multiple Testing

No Statistical Signifiance

Statistical Parameters

Mock Interviews

Amazon Testing Recommender System

Improve Driver Enrollments on Uber

Ab Testing

Last Updated:&nbsp;01/01/23

<h2 class="wp-block-heading">Problem</h2>

Duration: 20 Minutes Difficulty: Hard

An interviewer at&nbsp;Amazon&nbsp;asked:

Amazon’s search team wants to run an experiment that tests a new design of the search page along with a new product ranking algorithm. The new product search page shows fewer products per page and uses a more advanced algorithm for product ranking. How would you design an AB test?

<h2 class="wp-block-heading">Tips &amp; Hints</h2>

Last Updated : &nbsp; 01/01/23 Problem Duration : 20 Minutes Difficulty : Hard An interviewer at&nbsp; Amazon &nbsp;asked: Amazon’s search team wants to…

DataInterview

Multiple Testing

Problem#

Unlock the full lesson

Problem