Problem

Duration: 10 Minutes
Difficulty: Medium

An interviewer at Google asked:

An experiment has not been completed yet, but the p-value is less than alpha, the significance level. Should you launch the change?

Tips & Hints

This section provides tips and hints to help you practice this case problem, effectively. The practice tip section highlights how to practice this question. The solution hint section provides clues on how to solve this problem if you get stuck. Skip the solution hint if you want to try this problem on your own.

Practice Tip

To practice this problem effectively, set a timer for 10 minutes and explain your solution out loud to yourself or a practice buddy.
Make sure to use a whiteboard, sheet of paper, or word document to jot down your ideas. Jotting down your ideas will help structure your response.
Share your solution either on the course discussion or Slack channel to receive feedback from peers and instructors.
As you read the solution dialogue, sometimes you will see “*Address this question” after an interviewer asks a follow-up to the candidate. Pause the reading for a minute to respond to this follow-up as though you are the candidate.

Solution Hint

Consider the best practice in AB testing. Explain whether it is sound to evaluate a statistical result before the required sample size has been achieved.

Solution

[Interviewer] An experiment has not been completed yet, but the p-value is less than alpha, the significance level. Should you launch the change?

[Candidate] It’s not a good idea as that’s a classic AB testing mistake called peeking. This is when a decision is made prematurely before the desired sample size has been achieved. In an experiment, you typically set the statistical power, somewhere between 0.80 and 0.90 and estimate the sample size required to achieve the power. If you peek, there’s a chance that you made a decision based on the point estimate that’s far from the population mean.

[Interviewer] Can you elaborate on what you mean by “the point estimate that’s far from the population mean?”

[Candidate] Certainly, when the sample size is low, there’s high variability in the point estimate so the point estimate could fluctuate among positive, neutral and negative lifts while crossing the statistical significance threshold off and on. As a consequence, in one instance of the peeking, you might see that the statistical significance has been achieved with a positive lift. But, in another instance of peeking, the statistical significance may not have been achieved.

[Interviewer] Question, how does peeking affect the type 1 error rate of the experiment? *Address this question.

[Candidate] That’s a good question. Typically, the type 1 error rate, or the false rejection rate of the null hypothesis, is the same as the alpha, or the significance level. So, if the alpha is 0.05, then the false rejection rate is also 0.05. However, this calculation is based on the assumption that you make a decision after an experiment is complete, meaning that the required sample size is collected.

When you peek, there are multiple instances when you see that the p-value is less than alpha, leading you to believe that the effect has statistical significance when, in reality, there isn’t. The volatility in the p-value is because, with a low sample size, the point estimate has high variability given random chance. So, there are multiple instances when the point estimate of the treatment is extreme enough such that the p-value becomes less than alpha.

So, ultimately, when you peek in the midst of an experiment, you inflate the chance of committing the type 1 error rate.

[Interviewer] Now, what about in terms of the statistical power? *Address this question.

[Candidate] There is a relationship between the statistical power and sample size. As you increase the sample size, the statistical power increases. This is because when the sample size increases, the null and alternative distributions become narrower and convergent toward the population means.

Assessment

The candidate is assessed across two attributes: statistics and communication. Each attribute is rated based on the following scale:

Outstanding – Quick response with a sound solution.

Good – Minor mistakes, but converged toward a final solution that is sound.

Borderline – Required several hints before providing a sound solution.

Inadequate – Slow and incorrect response.

Based on the rubric above, the candidate receives the following remarks:

Statistics – Outstanding

The candidate received an outstanding remark given that:

The candidate understands that the point estimate is prone to high variability when the sample size is low. Therefore, even if the statistical significance is achieved early, the experiment should not stop until the required sample size has been acquired.
The candidate understands the relationship between the sample size, type 1 error rate, and statistical power.

Communication – Outstanding

The candidate received an outstanding remark given that he explained quickly, clearly, and comprehensively across the statistics questions.

Peeking at the P-Value

Problem

Tips & Hints

Unlock now by joining.