Last Updated: 08/21/21

Problem

Duration: 5 Minutes
Difficulty: Hard

An interviewer at Facebook asked:

An experiment showed that a UI change on the News Feed increased the click-through rate (CTR) on ads. However, given that the p-value is 0.06, should the change launch or not?

Tips & Hints

This section provides tips and hints to help you practice this case problem, effectively. The practice tip section highlights how to practice this question. The solution hint section provides clues on how to solve this problem if you get stuck. Skip the solution hint if you want to try this problem on your own.

Practice Tip

To practice this problem effectively, set a timer for 5 minutes and explain your solution out loud to yourself or a practice buddy.
Use a whiteboard, sheet of paper, or word document to jot down your ideas, which can help structure your response.
Share your solution either on the course discussion or Slack channel to receive feedback from peers and instructors.
As you read the solution dialogue, sometimes you will see “*Address this question” after an interviewer asks a follow-up to the candidate. Pause the reading for a minute to respond to this follow-up as though you are the candidate.

Solution Hint

A naive solution is to reject the launch just because the p-value is less than the significance level of the test. Do consider how practical significance plays a role in deciding a launch.
Structure the response by diagnosing the problem with key questions and providing guidance.

Solution

[Interviewer] An experiment showed that a UI change on the News Feed increased the click-through rate (CTR) on ads. However, given that the p-value is 0.06, should the change launch or not?

[Candidate] Thank you for the question. I know that News Feed, a core feature on Facebook, displays user posts and advertisements in an endless scroll. Before I provide a recommendation, can I ask clarifying questions?

[Interviewer] Certainly.

[Candidate] What was the significance level (alpha) of the test? If alpha = 0.05, then the p-value is greater than or equal to alpha, so the test failed to reject. This rejection means that the test should not launch.

Commentary: The candidate overlooked that the p-value is close to the alpha value and failed to ask a key question about the practical significance observed.

In general, the p-value is compared against alpha to determine whether the test effect is statistically significant. If p-value >= 0.05, do not reject the null hypothesis that the control and test are the same, and do not launch. Otherwise, reject the null hypothesis and launch.

However, the p-value is not the be-all-end-all. You also need to consider the practical significance. If the p-value is greater than, yet close to, the alpha, and the lift observed is high, launch the change with caution.

Ultimately, a go-or-no-go decision should be based on:

P-Value vs Alpha
The magnitude of the practical significance

[Interviewer] What if the product manager still wants to launch? What would you do then? *Address this question.

[Candidate] I would discourage the product manager from launching as the lift observed may not generalize beyond the experiment. In fact, I would encourage the manager to re-run the experiment with a larger sample size to increase the statistical power, the probability of detecting an effect given that the effect exists.

Commentary: The response that the candidate provided is incomplete. The candidate should have assessed the situation before providing a recommendation that the product manager may not agree on.

The candidate should have asked the interviewer clarifying questions such as:

What is the lift observed from the baseline?
What is the potential risk to News Feed if the change has no effect?
Is the UI change major or minor?

The first question explores the practical significance of the effect observed. As the lift increases, more support is provided in launching.

The second question explores the potential risk if a change with no effect is rolled out. Does the PM anticipate high or low risk to the user engagement and monetization on News Feed? If the risk is deemed low, then regardless of the statistical significance, the feature should launch.

The last question delves into the engineering effort required to launch and maintain the change. If the change is minor, then this supports PM’s decision.

As you can see, exploring the situation, then providing guidance, is the best way to approach this question.

Assessment

The candidate is assessed across two attributes: AB testing and communication. Each attribute is rated based on the following scale:

Outstanding – Quick response with a sound solution.

Good – Minor mistakes, but converged toward a final solution that is sound.

Borderline – Required several hints before providing a sound solution.

Inadequate – Incorrect response.

Based on the rubric above, the candidate receives the following remarks:

AB Testing – Borderline

The candidate failed to see the subtlety in deciding whether to launch based on key details provided. For instance, the candidate naively assumed that p-value is the be-all-end-all in a launch decision. However, as noted in the commentary, considering practical significance is also vital. In the follow-up question, the candidate jumped to a conclusion without diagnosing the situation. Asking key questions then forming a sound recommendation is important in AB testing. Therefore, the candidate could not receive a rating beyond a good rating.

Though the candidate provided generic suggestions, the ideas were not too far-fetched. Given this, the candidate receives a rating that is borderline, not inadequate.

Communication – Good

Overall, the candidate explained her ideas clearly and concisely such that a good rating was justifiable. However, she failed to receive an outstanding remark given that she failed to ask key questions to provide sound recommendations. For instance, on the last question on guiding the PM, instead of assessing the situation first, she jumped to the conclusion that the change should not launch just because the p-value was less than alpha.

[Meta] P-Value > Significance Level (Alpha)

Problem

Unlock now by joining.