Introduction

Downloadable Code Files

Getting Started with Code Files

Understand Business Problem

Experiment Goals

Key Members

Product KPI

The North Star Metric

Driver Metric

Guardrail Metrics

Secondary Metrics

Segmentation Metrics

Launch Criteria

Experiment Data

Amazon AB Testing: Question

Amazon AB Testing: Solution

Code Walkthrough

State the Experiment Hypothesis

Hypothesis Statement

Hypothesis Statement: Question 1

Chi-Squared Test of Independence: Question

Hypothesis Statement: Solution 1

Hypothesis Statement: Question 2

Hypothesis Statement: Solution 2

Significance Level

Significance Level: Question

Significance Level: Solution

Statistical Power

Statistical Power: Question 1

Statistical Power: Solution 1

Statistical Power: Question 2

Statistical Power: Solution 2

Minimum Detectable Effect (MDE)

Minimum Detectable Effect: Question

Minimum Detectable Effect: Solution

Design the Experiment

Randomization Unit

Sample Size Calculation

Sample Size Calculation: Question

Sample Size Calculation: Solution

Experiment Duration

Experiment Duration: Question

Experiment Duration: Solution

Run the Experiment

Experiment Platform

Peeking at P-Values

Assess Validity Threats

Stable Unit Treatment Value Assumption

Survivorship Bias

Sample Ratio Mismatch

Primacy Effect

Novelty Effect

Holiday Effect

Other Effects to Consider

AA Test

Conduct Statistical Inference

Chi-Squared Test of Independence

Chi-Squared Test of Independence: Solution

Z-Test for Proportions

Z-Test for Proportions: Question

Z-Test for Proportions: Solution

T-Test for Means

T-Test for Means: Question

T-Test for Means: Solution

Launch Decision

Decide Whether to Launch

Interpret Test: Relative Difference

Launch Decisions

Decision Rule

Confidence Interval

Holdout Set

Practice Cases

Peeking at the P-Value

Failed to Generalize

Multiple Testing

No Statistical Signifiance

Statistical Parameters

Mock Interviews

Amazon Testing Recommender System

Improve Driver Enrollments on Uber

Ab Testing

<h2 class="wp-block-heading">Problem</h2>

Duration: 10 Minutes Difficulty: Medium

An interviewer at&nbsp;Google asked:

An experiment has not been completed yet, but the p-value is less than alpha, the significance level. Should you launch the change?

<h2 class="wp-block-heading">Tips &amp; Hints</h2>

This section provides tips and hints to help you practice this case problem, effectively. The&nbsp;practice tip&nbsp;section highlights how to practice this question. The&nbsp;solution hint&nbsp;

Problem Duration : 10 Minutes Difficulty : Medium An interviewer at&nbsp; Google asked: An experiment has not been completed yet, but the p-value is less…

DataInterview

Peeking at the P-Value

Problem

Tips & Hints

Unlock the full lesson

Peeking at the P-Value

Problem#

Tips & Hints#

Unlock the full lesson

Problem

Tips & Hints