Last Updated: 01/01/22

Problem

Duration: 5 Minutes
Difficulty: Easy

An interviewer at Instacart asked:

1. How do you determine the significance level (alpha) of a statistical test?

2. How do you determine the power of a test?

3. How would you determine the minimum detectable effect (MDE) of a test?

Tips & Hints

This section provides tips and hints to help you practice this case problem, effectively. The practice tip section highlights how to practice this question. The solution hint section provides clues on how to solve this problem if you get stuck. Skip the solution hint if you want to try this problem on your own.

Practice Tip

To practice this problem effectively, set a timer for 5 minutes and explain your solution out loud to yourself or a practice buddy.
Use a whiteboard, sheet of paper, or word document to jot down your ideas, which can help structure your response.
Share your solution either on the course discussion or Slack channel to receive feedback from peers and instructors.
As you read the solution dialogue, sometimes you will see “*Address this question” after an interviewer asks a follow-up to the candidate. Pause the reading for a minute to respond to this follow-up as though you are the candidate.

Solution Hint

Avoid providing generic responses such as 0.05 for the significance level, 0.80 for the statistical power and 1% for MDE. Though these are values often seen in online experiments, the interviewer is assessing how you explore which value to choose given a range and justify your choice.

Solution

[Interviewer] I’d like you to discuss how you decide the significance level (alpha) of a statistical test.

[Candidate] The usual industry standard is 0.05. So, I would say that this is a reasonable value for the significance level.

Commentary: The candidate lacks substance with her response. It is true that, commonly, the significance level is set at 0.05 in AB testing. However, merely, saying that 0.05 should be chosen because it’s the “industry standard,” ignores the trade-offs associated across the range of significance levels. Note that the significance level is associated with the type 1 error rate and statistical power. Choosing and justifying the significance level you chose for a test is the key to acing this interview question.

Consider this range chart:

Significance level	When to choose this value
0.01	If the risk of the type 1 error rate is too high, then reduce the significance level to 0.01. The risk of a false-positive is often calculated during the hypothesis stage before executing the experiment. Assessing the risk in advance helps identify an appropriate significance level. For instance, suppose that Amazon wants to test a new checkout flow that could increase millions of dollars in monthly revenues should it succeed. However, the risk of a false-positive could be higher in $ the cost of building, experimenting and rolling out a feature that, in reality, has no lift. In other words, all the effort was put into testing and building a feature for wider use to no avail.
0.05	You choose 0.05 as the significance level if the feature you test is innocuous or you expect a slightly positive lift. For instance, suppose you test a new layout on the landing page of a website. You are not necessarily expecting a lift from the new layout, but it’s just nice to have. In this case, setting the significance level as 0.05 is reasonable.
0.10	If you hypothesize that the feature will bring a highly positive impact, then you accept a higher type 1 error rate and run an experiment with it. This significance level is set when the business and product teams strongly hypothesize that the new feature will bring a solid positive impact. Therefore, they are willing to accept the higher risk of the type 1 error rate to launch.

As you can see in the range above, the significance level is chosen based on a cost-and-benefit analysis of launching or not launching a test. This analysis is performed upfront in the hypothesis stage which ultimately drives which significance level to choose for running an experiment.

[Interviewer] Okay, what about the statistical power? How would you choose this for an experiment? *Address this question

[Candidate] I would say that choosing statistical power depends on various circumstances. The typical range of the statistical power is between 0.70 and 0.95, with 0.80 being the most common statistical power.

I would consider that there are trade-offs associated with each value of the statistical power in the range. As the statistical power decreases, the probability of detecting an effect when it exists decreases. Simultaneously, the sample size decreases, which increases the variability of the point estimate which could lead to bias estimation of the lift observed in the experiment. The higher the sample size means that you can acquire an estimate close to the population parameter.

On the other hand, if you increase the sample size, you increase the probability of detecting an effect when it exists. Simultaneously, your sample size increases, which reduces the variability; thereby, acquiring an estimate close to the population parameter. The only downside is that, since you require a large sample size unless you already have a large mass of user traffic at your disposal, getting your experiment result may take a long time.

Commentary: The candidate provided a comprehensive response to choosing the statistical power. Note that she didn’t just say 0.80 given that it’s industry standard. There are subtleties to choosing the statistical power. In this case, the candidate explained how the statistical power is associated with the sample size requirement, parameter variability, and experiment duration.

The only part she left out in her response is how she would form a recommendation based on the subtleties of choosing a statistical power. See the chart below:

Statistical Power	When to choose this value
0.7	If the user traffic available is low, and you may not want to wait a long time to collect the sample, then lower the statistical power.
0.8	This is a reasonable statistical power to start with. You can start off with this value first, and decide on the next step after the experiment. As described in the 0.90 – 0.95 range, if the test failed, but you have a strong belief that it succeeds, then re-run with a higher statistical power.
0.90 – 0.95	If you are re-testing an experiment that failed to achieve statistical significance the first time. Suppose that, in an experiment, the p-value of a test was 0.06, the alpha was set at 0.05, and the statistical power was set at 0.80. The point estimate is moderately positive (you expected a 5% relative lift, and you observed a 4% relative lift). Given that there’s a 20% chance (Type II Error Rate = 1 – Power) of failing to detect an effect when there is one, you reduce the error rate by increasing the power in a re-run. This re-run is done if you have high confidence that the new feature works.

[Interviewer] Lastly, how would you choose the MDE of a statistical test? *Address this question

[Candidate] I would say that the MDE is 1% which is common in online platforms with millions of users, like Instacart.

Commentary: Similar to the response provided with the significance level, the candidate failed to explain the subtleties of determining an MDE. The MDE should not be set at 1% just because it is deemed as the industry standard.

It is chosen based on what the business considers to be practically significant.

Consider the following scenario. Suppose that an online store currently generates $100M per year given the baseline conversion rate of 10%. If the MDE was set at the 1% lift, you expect the new conversion rate from a new feature to be at 11%, which is equivalent to $10M in additional revenue. This seems to be a significant boost to the revenue, but it doesn’t weigh in on the cost of experimenting and maintaining the new feature. Hence, one needs to consider the return on investment (ROI). The money that’s invested should achieve a higher return later. In this example where the return is $10M, if the cost is less than the return, then the business can consider the 1% lift as practically significant. If the cost is equal to or greater than the return, then the 1% lift is too low.

Here’s another chart to consider when determining an MDE.

Minimum Detectable Effect	When to choose this value
1% to 5%	Consider this range in a large-scale online platform that already generates billions, if not hundreds of millions, of dollars in revenue per year. Any incremental gain in a key metric could equate to millions of dollars in additional revenue. An important note is that you do want to consider the ROI and adjust the MDE accordingly as mentioned earlier.
5% to 10%	Choose this range if the user size available is too low and/or the gain needs to be high to justify a change. This MDE could be seen in start-ups or new flagship products of a large company that expects fast-paced growth.

Assessment

The candidate is assessed across two attributes: AB testing and communication. Each attribute is rated based on the following scale:

Outstanding – Quick response with a sound solution.

Good – Minor mistakes, but converged toward a final solution that is sound.

Borderline – Required several hints before providing a sound solution.

Inadequate – Incorrect response.

Based on the rubric above, the candidate receives the following remarks:

AB Testing – Borderline

The candidate provided a comprehensive response to how he would choose the statistical power. He didn’t just say that he would set the statistical power as 0.80 just because it’s a common value set in experiments. He understands how the sample size, the variability of the sample size and the experiment runtime vary given the power. One suggestion to him is that he should explain how he would provide a recommendation. For instance, given that the range could be somewhere between 0.70 and 0.95, how would he decide which value to choose based on what condition? More clarity on how he would recommend would be helpful.
In both questions on the significance level and MDE, he failed to explain the subtleties of setting the values most appropriate given various conditions. In both questions, he merely deferred to the values commonly set in experiments. This shows that he is inexperienced in experiments, and he doesn’t understand the reasoning behind how the significance level and MDE should be set.

Communication – Borderline

Though the candidate provided a clear response when explaining how he would choose statistical power, the details of his response were often lacking in follow-up questions involving the significance level and MDE.

[Instacart] Statistical Parameters in AB Testing

Problem

Unlock now by joining.