Last Updated: 01/01/23

Problem

Duration: 10 Minutes
Difficulty: Medium

An interviewer at Amazon asked:

Amazon tested a new feature that displayed “frequently bought together” on the product page. For instance, when a user is on the product page of a lamp, the feature displays LED lightbulbs that other users have also bought. The experiment shows an improvement.

Compared to the baseline which does not display related products, the feature showed an improvement in the test. However, a few weeks after the launch to the rest of the users, the performance reverted to that of the baseline. What happened?

Tips & Hints

This section provides tips and hints to help you practice this case problem, effectively. The practice tip section highlights how to practice this question. The solution hint section provides clues on how to solve this problem if you get stuck. Skip the solution hint if you want to try this problem on your own.

Practice Tip

To practice this problem effectively, set a timer for 5 minutes and explain your solution out loud to yourself or a practice buddy.
Use a whiteboard, sheet of paper, or word document to jot down your ideas, which can help structure your response.
Share your solution either on the course discussion or Slack channel to receive feedback from peers and instructors.
As you read the solution dialogue, sometimes you will see “*Address this question” after an interviewer asks a follow-up to the candidate. Pause the reading for a minute to respond to this follow-up as though you are the candidate.

Solution Hint

This style of question is similar to the ones asked in the metric investigation in product sense rounds. The business and product team expects the metric to move in a certain direction. But, the result is the opposite. A combination of internal and external factors could contribute to this. Consider why.

Solution

[Candidate] Thank you for the question. I assume that the primary metric of the experiment is the average revenue per user. To start the diagnosis of the problem, I have questions about the experiment design. What was the statistical power and duration of the experiment?

[Interviewer] The statistical power was set at 0.80, and the experiment ran for two weeks. Why do you ask?

[Candidate] The reason I ask is the following. If the statistical power is too low, the sample size is low, meaning that the sample mean has high variability. The high variability of the sample means could mean that the experiment could achieve statistical significance prematurely before it converges toward the true mean. Since the statistical power is set at 0.80, I believe this is reasonable. So, I will rule out the possibility that the low power could have led to the incorrect decision to launch. In terms of the experiment duration, it’s best to run the experiment for at least a week to eliminate the day-of-the-week effect. If the experiment ran over the weekend, the order value might increase, but the result could fail to stay consistent during the weekdays. Given that the experiment ran for two weeks, I can also eliminate the possibility that the effect has on the outcome.

[Interviewer] Good explanations. Can you think of any other factors that could affect the outcome? *Address this question.

[Candidate] I believe there’s another possibility that the experiment result happened by chance. Quite simply, the type 1 error rate had occurred. If the alpha was set at 0.05 and you repeated the experiment 100 times, about 5% of the experiments will reject the null even though it’s true. We can re-run the experiment. But, before we do this which consumes time, we can look into other factors.

[Interviewer] Okay, what would those be?

[Candidate] I would check the internal validity of the experiment. Perhaps, there were issues with the randomization, bugs with the UI and e.t.c. that could have contaminated the experiment result. If the checks are sound, then we can look into external factors. Maybe, the competition launched discounts on the day that the new version was rolled out. Customers might have switched over to the competitor’s website.

[Interviewer] Okay, if the competition played a role, how would you guide the product manager on how to proceed? *Address this question.

[Candidate] I would look at the duration of the coupon campaign on the competitor’s website. If it’s less than a week, I would say wait it out. When their campaign ends, we may see the performance of the new version climb back up to the level that was seen in the experiment. If the result is not seen, then there’s a bigger conversation at play. Assuming that no other factors are causing the discrepancy, and the competition is the root cause, Amazon may need to address this at the business level and strategize how to get users back onto the website as a change on the checkout page alone will not restore the improvement.

[Interviewer] Can you think of any other factors that could cause the discrepancy?

[Candidate] I can also see how the holiday effects could cause the discrepancy. Perhaps the lift was seen due to the holiday effect from Christmas day. After the shopping season ended, the holiday effect dissipated and; ultimately led to the performance of the test falling to that of the control. Lastly, I could see how the novelty effect could affect impact this.

[Interviewer] Interesting, how would the novelty effect compromise the experiment result? *Address this question.

[Candidate] Well, the novelty effect means that users reacted positively to the change in the beginning. Eventually, the novelty wears off, and the test performs at the same level as the baseline. I can see if this is the case by analyzing how the experiment was performed between new and recurrent users. If novelty played a role, then we would see that recurrent users reacted positively to the website change while the new users remained neutral in comparison to the cohorts in the control condition.

Assessment

The candidate is assessed across two attributes: AB testing and communication. Each attribute is rated based on the following scale:

Outstanding – Quick response with a sound solution.

Good – Minor mistakes, but converged toward a final solution that is sound.

Borderline – Required several hints before providing a sound solution.

Inadequate – Incorrect response.

Based on the rubric above, the candidate receives the following remarks:

AB Testing – Outstanding

The candidate understands how an external validity threat could come from a competitor and how to assess whether this has a factor in the experiment.
The candidate demonstrated that he understands the checkout feature and the key metric that is being tested.
The candidate understands how faulty experiment design (i.e. low power and duration) could affect the outcome of the test.
The candidate provided a comprehensive list of internal and external validity checks he would perform to identify the root cause of the inconsistent results.

Communication – Outstanding

The candidate articulated his questions and solutions clearly.
The candidate ensured that he understood the problem by asking questions.

Failed to Generalize

Problem

Unlock now by joining.