Last Updated: 01/01/23

Problem

Duration: 5 Minutes
Difficulty: Hard

An interviewer at Amazon asked:

Amazon’s search team wants to run an experiment that tests a new design of the search page along with a new product ranking algorithm. The new product search page shows fewer products per page and uses a more advanced algorithm for product ranking. How would you design an AB test?

Tips & Hints

This section provides tips and hints to help you practice this case problem, effectively. The practice tip section highlights how to practice this question. The solution hint section provides clues on how to solve this problem if you get stuck. Skip the solution hint if you want to try this problem on your own.

Practice Tip

To practice this problem effectively, set a timer for 5 minutes and explain your solution out loud to yourself or a practice buddy.
Use a whiteboard, sheet of paper, or word document to jot down your ideas, which can help structure your response.
Share your solution either on the course discussion or Slack channel to receive feedback from peers and instructors.
As you read the solution dialogue, sometimes you will see “*Address this question” after an interviewer asks a follow-up to the candidate. Pause the reading for a minute to respond to this follow-up as though you are the candidate.

Solution Hint

Define the Key Metric of the Experiment – What should be the primary metric evaluated in the experiment? Should it be the # of product pages viewed? Items ordered? Items purchased?
Design an Experiment – How would you design the experiment that tests the changes?
Decisioning – How would you decide whether to launch the changes or not?

Solution

[Interviewer] Amazon is testing a new ranking algorithm along with a new UI that improves the search result. How would you design an experiment to test the changes?

[Candidate] Thank you for the question. I have few clarifying questions to ask to design an experiment. First, what is the metric that is being evaluated in the experiment? Secondly, is the experiment testing both the ranking algorithm and UI, simultaneously?

[Interviewer] On your first question, you will need to define the metric. On your second question, yes, both algorithms are being tested. Note that you need a way to isolate the effects coming from the ranking algorithm and the UI.

[Candidate] Okay, thank you for the clarification. I can start with a metric to evaluate in the experiment. First, I need to consider the user experience and identify the key user action that the test should increase.

I know that, on Amazon, a user can query for a product on the search bar, and a list of products is displayed in the search result. The list is most likely ranked based on the highest propensity of purchase to the lower propensity. The new UI + algorithm should increase purchases given that as users find more relevant items. As an effect, Amazon’s revenue will increase.

So, I believe the key metric to evaluate is the average number of items purchased per user.

Commentary: The candidate is on the right track in providing the primary metric to evaluate in the experiment. He started by demonstrating product sense on Amazon’s key feature, then worked toward a metric defining and success.

However, a few things are lacking in the final metric. The primary goal indeed is to increase revenue through purchases. In that case, the candidate should have tracked the average purchase value ($) per user, not the average number of items purchased per user. The reason is that the former metric is more correlated with Amazon’s KPI in increasing sales revenue given that the metric measures $. When determining the primary metric for AB testing, think about the north star metric of the business, and either use this metric in the AB testing or identify a short-term metric that serves as a proxy for the long-term metric.

[Interviewer] Okay. In your metric, I noticed that you are using “per user” as the unit of analysis. How would you account for multiple observations coming from the user? Let’s say that a user in the treatment has multiple sessions. Are there any corrections you want to consider? *Address this question

[Candidate] I understand what you are conveying. I know that most statistical tests such as the T-Test assume that the observations are independent. If there are multiple observations collected per user, the assumption is violated. To correct this, just averaging across multiple observations from the user will work. This will allow you to have one observation per user.

Commentary: The candidate is correct about the observation independence is being violated. But, the correction is flawed.

The reason is that “averaging” across multiple observations still means that the test is collecting multiple observations per user. To correct this, some constraints could be applied. One approach is by only measuring the first user session once the user enters the experiment. The other aspect is to constrain by time. Collect the total purchase value over 1 day or 1 week upon entering the experiment. The constraints allow only one observation collected per user.

[Interviewer] Okay. Now, how would you design your experiment? *Address this question

[Candidate] I would start with the following hypothesis:

Ho: Old UI + Old Algorithm

Ha: New UI + New Algorithm

I would set the statistical significance at 0.05, power at 0.80, and MDE at 1% relative lift, and use the T-Test for comparing the means between the control and treatment. Based on the parameters, I can determine the sample size and allocate enough traffic to run the experiment for 1 to 2 weeks.

Commentary: The candidate failed to see the subtle. The New UI and algorithm are two separate changes, each with its effects. When you run a test with both only, you are unable to measure whether the observed effect is from the UI, the algorithm, or the combined effects of the two.

Therefore, an improved design would consider multiple comparisons:

Ho: Old UI + Old Algorithm

H1: New UI + Old Algorithm

H2: Old UI + New Algorithm

H3: New UI + New Algorithm

Using the pairwise T-Test comparing Ho vs H1, Ho vs H2, and Ho vs H3, would help isolate the effects on both changes. This is critical for deciding which change to launch. For instance, if it turns out that the new algorithm alone brings a higher lift than the new UI and algorithm combined, then it makes more sense to just launch with the new algorithm alone for higher lifts while reducing the costs of changing both.

[Interviewer] Okay, now how would you determine whether to launch or not? *Address this question

[Candidate] If P-value is less than the significance level at 0.05, then ramp up to the higher % of users who are exposed to the changes that include the new UI + algorithm. Once the effect observed remains consistent from one stage of the ramp to the next, roll out the changes to the changes to the rest of the user base. This also assumes that both internal and external validities of the experiment are sound.

Assessment

The candidate is assessed across two attributes: AB testing and communication. Each attribute is rated based on the following scale:

Outstanding – Quick response with a sound solution.

Good – Minor mistakes, but converged toward a final solution that is sound.

Borderline – Required several hints before providing a sound solution.

Inadequate – Incorrect response.

Based on the rubric above, the candidate receives the following remarks:

AB Testing – Borderline

Recognized that a success metric needs to be defined for the experiment. This is a prerequisite before AB testing is designed.
Understands statistical parameters in designing an AB test (i.e. significance level, power, MDE, experiment duration).
Failed to recognize the correct unit of analysis when proposing a metric. The observation window per user needs to be capped by either the session sequence (i.e. the first session) or time (i.e. $ purchase within 24 hours of entering the experiment).
Failed to set up a sound hypothesis. Instead of one treatment Ha: New UI + New Algorithm, there should be a total of three. H1: New UI + Old Algo, H2: Old UI + New Algo, and H3: New UI + New Algo. These ensure that effects could be correctly isolated.
Demonstrated some sense of how to decide whether to launch or not based on experiment results (i.e. p-value vs significance level, ramp-up and internal & external validity checks).

Communication – Outstanding

Asked clarifying questions to brainstorm (i.e. “What is the metric that is being evaluated in the experiment?” A naive response would have proposed an AB testing design without analyzing the problem statement first.
Explained his thought process clearly and comprehensively.

[Amazon] AB Testing Multiple Changes

Problem

Unlock now by joining.