Causal Inference

Downloadable Code Files

Getting Started with Code Files

Getting Started

Introduction

Experiment Goals

Key Members

Product KPI

The North Star Metric

Driver Metric

Guardrail Metrics

Secondary Metrics

Segmentation Metrics

Launch Criteria

Experiment Data

Amazon AB Testing Case - Question

Amazon AB Testing Case - Solution

Code Walkthrough

Understand the Business Problem

State the Experiment Hypothesis

Hypothesis Statement

Significance Level

Statistical Power

Minimum Detectable Effect (MDE)

Design the Experiment

Randomization Unit

Sample Size Calculation

Experiment Duration

Run the Experiment

Experiment Platform

Peeking at P-Values

Assess Validity Threats

Stable Unit Treatment Value Assumption

Survivorship Bias

Sample Ratio Mismatch

Primacy Effect

Novelty Effect

Holiday Effect

Other Effects to Consider

AA Test

Conduct Statistical Inference

Chi-Squared Test of Independence

Z-Test for Proportions

T-Test for Means

Decide Whether to Launch

Interpret Test Results (Relative vs Absolute Difference)

Launch Decisions

Decision Rule

Decision Rule with Confidence Interval

Holdout Set

[Google] An experiment is statistically significant, but the sample is low. Should you change?

[Amazon] An AB test on "frequently bought together" showed improve in test, but not after launch. What happened?

[Amazon] New UI and algorithm are tested on product search. How would you design an experiment?

[Facebook] An experiment on News Feed showed increase in CTR, but the p-value is 0.06. Should you launch the change?

[Instacart] What are important statistical parameters in AB testing?

Interview Cases

[Amazon] Testing Recommender Systems

[Uber] Improving Driver Enrollment Rate

Mock Interviews

Go beyond A/B testing and delve into the essential frameworks of causal inference that are asked in experimentation interviews at product companies (e.g. Google, Uber, DoorDash). This course offers a deep dive into key methods such as Propensity Score Matching (PSM), Regression Discontinuity Design (RDD), Difference in Differences (DiD), and Synthetic Controls. These methods are crucial for measuring causal relationships in observational data. Whether you're preparing for interviews at top tech companies or strengthening your data science fundamentals, this course will guide you through advanced frameworks used to evaluate causality and ace interviews