Problem

Netflix wants to introduce a new movie recommendation algorithm to increase the long-term retention of subscribed users. How would you run an experiment?

Tips

To address this product case question, structure the response in the following manner:

Ask clarifying questions – Get clarity on the “long-term retention” measured.
Define metric(s) – What’s a reasonable metric for the long-term retention of users?
Design the experiment – How would you design this experiment?
Provide a recommendation – What is your final recommendation to a business stakeholder?

Solution

[Interviewer] Netflix wants to introduce a new video recommendation algorithm to increase the long-term retention of subscribed users. How would you run an experiment?

[Candidate] Thank you for the question. I can sense the Netflix wants to increase retention as this means an increase in revenue from the monthly subscription. Now, I have a clarifying question. When you measure the long-term retention, do you already have a metric?

[Interviewer] Let’s say that we do not have one yet, and you need to recommend one.

Commentary: Notice that the candidate did not assume that he understands how the retention metric is measured. When asked an open-ended question, it’s vital to come to terms with the interviewer. For instance, when an interviewer asks questions such as “how would you measure the success of a product launch?”, “How would you measure the satisfaction of Uber riders?,” you need to find metrics that serve as proxies for those qualitative experiences.

[Candidate] You mentioned that the retention you care about is the “long-term” retention. Usually, in products, long-term metric measures active engagement after one year. And, the active engagement could be the user subscription. Therefore, I believe a reasonable long-term metric is the % of users who remain subscribed after 12 months since sign-up.

Commentary: Note that that the cutoff that determines an active user in the long term is arbitrary in this problem. The main point is that the candidate understands how to create a metric that is specific.

[Interviewer] Okay, what’s the next step in the process?

[Candidate] Now, I need to design an experiment that evaluates whether the long-term retention increases given the new recommendation algorithm. I’d like to start with the following hypothesis:

Long-Term Retention: % of users who remained subscribed after 12 months since sign-up.

Ho: The long-term retentions are the same for the current and new algorithms.

Ha: The long-term retentions are different for the current and new algorithms.

After defining the hypothesis statements, I can consider the statistical parameters in the experiment:

Significance level: 5%

Power: 80%

Experimentation runtime: 1-2 weeks

Users sampled: Global

Statistical testing: T-Test for proportions

[Interviewer] Quick question. If it takes you 12 months to measure the long-term retention, wouldn’t this be a problem given that your experimentation runtime is just 1-2 weeks?

Commentary: The interviewer posed a trick question that many candidates get stumped on. Note that the “experimentation runtime” needs to accommodate for the time it takes to find users and collect behavioral data. Given that Netflix has over 100+ million active users, finding enough users to test is not an issue. However, the time it takes to collect their retention 12 months later could pose an issue. More on this in the next commentary.

[Candidate] I believe 1-2 weeks should be sufficient in gathering enough users to run the experiment. Once they are enrolled in the experiment as either the control or treatment group, we can monitor their retention 12 months later.

Commentary: Waiting 12 months for an experiment to complete is too long for a large-scale online platform that values quick testing and decisions. The best way to mitigate this is to find a short-term metric that serves as a proxy for the long-term metric.

Based on historical data, find a short-term metric (i.e. Hours spent on videos per week) that correlates the most with 12+ month retention. Use this short-term metric as the key metric in the experiment. This short-term metric can be found using statistical correlation techniques such as the Spearman or Pearson correlation.

Essentially, the revised experiment uses this metric to compare:

Short-Term Metric (Proxy for long-term retention): Average # of hours spent on Netflix per user in one week

Interviewer Assessment

The candidate is assessed based on three criteria: business/product sense, statistical methodology, and communication. Each dimension contains the following rating: (1) not competent, (2) marginal, (3) adequate, (4) good, (5) superior.

Business and Product Senses – 4

The candidate receives a good remark based on the following instances:

The candidate ensured that he defined the retention metric quantitatively. A subscription beyond 12 months as a proxy for long-term retention is reasonable.
Understanding the business needs of an experiment is vital. Though the candidate demonstrated strengths in detailing the statistical parameters of the experiment, he failed to recognize that the experiment should be completed quickly.

Statistical Methodology – 2

The candidate receives a marginal remark based on the following reason:

Though the candidate understands the statistical parameters for setting up an experiment. The overall experimentation design was unsound. He failed to conjure up a design that tests the effect of the new algorithm on long-term retention. The candidate should accommodate potential business needs. Typically, in the experimentation cycle, an experiment should be completed within 1 to 2 weeks, and decisioning on the next step is made within a week or two after the experiment completion.

Communication – 5

The candidate receives a superior remark based on the following:

He asked questions to ensure that he understood the problem before diving into a solution.
The candidate explained his approach in a clear manner.

[Netflix] Long-Term Metric

Problem

Tips

Unlock now by joining.