Problem

Duration: 20 Minutes
Level: Senior
Area: Product
How would you incentivize inactive users to return to Uber Eats?

Hint

If you are stuck on this question, use the following framework to formulate a response:

Clarifying Questions
- What does an “inactive” user mean on Uber Eats

Business and Product Contexts
- What’s the main motive for getting inactive users to return on the platform?
- What are product strategies for incentivizing inactive users to return on the platform?
- What’s the main KPI on Uber Eats?

Experimentation Methodology
- What’s the primary metric to evaluate?
- What is your hypothesis you want to test?
- How would you run your experimentation?

Decisioning and Recommendation
- How would you interpret your analysis?
- What would be your recommendation to the product lead?

Candidate Example

[Interviewer] How would you incentivize inactive users to return to Uber Eats? *Try this question for 5 Minutes

[Candidate] Thank you for the question. To address this question, I think I need to first understand the meaning of “inactive users.” Can I presume that there’s already a label that defines whether a user is inactive or not?

Commentary:
Notice that the user first asks questions to remove ambiguity. Clarifying first before diving into solutions is vital in case problems.

[Interviewer] Good question, you will need to define what it means for a user to be inactive.

[Candidate] Understood. Usually when it comes to churn, this is defined by a state of inactivity in terms of no activity (i.e. logins, orders) for X period of time. And, depending on what is considered a regular usage of the product, the time window of inactivity that defines a churned user can be different. I would say that, for Uber Eats, a user has churned based on this criterion:

[While explaining, the candidate types notes on the Coderpad]

#Coderpad
Churned User = No orders the past 7 days

[Interviewer] How would you accommodate for the fact that a user may comeback on the platform after 7 days? Should such users still be labeled as churned? *Try this question for 3 Minutes

[Candidate] Good question. If the user returns and orders on the platform after 7 days of inactivity, then they should be relabeled as active. Ideally, we would want the churn window to be longer to be more certain that they have churned. For instance, we would be more certain that a user has left the platform permanently if they remained in active for 3 months vs 7 days. However, the risk is that if we wait that long (3 months) for the user to be labelled as churn, we risk not doing anything for 3 months to get the users to return to the platform. Hence, 7 days will be ideal as a proxy that users have churned regardless of whether some users intend to return to the platform on their own.

Commentary:
The candidate provided a decent analysis on churn. The determining “active” and “churned” users might seem simple at first, but the evaluation requires various considerations of the business KPI, business model and user behaviors.
In the case of Uber Eats, given the frequency of how often people eat (~ 21 meals per week = 3 meals per day x 7 days), it makes sense that Uber Eats would want, at least one of those meals is purchased on the app per week. You would expect a similar behavior on their commuter app. For additional practice, try to define active and churned users on other platforms such as Airbnb, Amazon, Netflix, Twitter, and Facebook.

[Interviewer] Great. Now, why do you consider orders, not logins, as the churned user? *Try this question for 3 Minutes

[Candidate] This is because it would better align with Uber Eats’ primary metric, which isn’t the number of login sessions but orders per some window. User logins do not generate revenue, but orders do. So, aligning with the primary business KPI, it would make sense for the inactivity status be based on orders, not logins.

[Interviewer] Okay, great. Now, suppose you have the labels. Now, what’s the next step? *Try this question for 3 Minutes

[Candidate] The next step would be to brainstorm a solution to get users to return to the platform and run A/B testing on the idea.

[Interviewer] Okay, what idea would you want to test? *Try this question for 3 Minutes

[Candidate] I’d like to consider a few options then choose one idea to test in A/B testing:

[While explaining, the candidate types notes on the Coderpad]

#Coderpad
Incentive: A discount – 15% off on the next order
Idea 1: Send an email with a discount for the next order
Idea 2: Send a phone notification with a discount for the next order
Idea 3: Display a popup with a discount when a user is on the app

I would need to choose the appropriate channel to test the idea that giving an incentive in the form of a discount (15% off on the next order) will get users to return on the platform. Since each idea requires separate engineering and design efforts to build and test, ideally, it’s best to choose one idea to test first. *Which one would you test first? Try this question for 3 minutes.

I’d like to use the process of elimination to choose one idea to test. First of all, idea 3 might be problematic as, among the churned users, those who did not visit the platform at all will not see the discount. So, this is problematic. Secondly, unlike the first two ideas which require minimal design and coding efforts, creating a popup on the app requires such efforts. Given these reasons, I would not test idea 3. Now, in terms of whether idea 1 or 2 should be tested, I would want to look at the statistics for each channels based on past experimentations: (1) # of views, (2) CTR, and (3) bounce rate. Based on the statistics, I would choose the idea to test that generally have the highest # of views and CTR and the lowest bounce rate.

The main takeaway here is to first find the channel that would result in the highest ROI. Then, iteratively optimize in other channels.

[Interviewer] Great, now how would you run an A/B testing on idea 1?

[Candidate] I need to start by formulating hypothesis statements to test:

[While explaining, the candidate types notes on the Coderpad]

#Coderpad
Ho: No intervention on churned usersH1: Send an email with a simple reminder: “Order on Uber Eats.”H2: Send an email with a discount: “Get 15% off on your next order on Uber Eats.”

*What do you believe is the intention for H1? Why not just include Ho and H2? Try this question for 3 minutes.

We want to include H1 as we want to see if the email alone has an effect on churned users. I also need to define the primary metric to track.

*What do you think should be the primary metric for this A/B test? Try this question for 3 minutes.

I am considering two metrics:

[While explaining, the candidate types notes on the Coderpad]

#Coderpad
Metric 1: % of churned users who ordered within a weekMetric 2: % of churned users who ordered the second time within a week after the first order.

If a churned user returns and orders during the observation period (7 days) then that might appear to be a success. However, that doesn’t mean that the user is now active again. The user might have comeback just to use the discount then not return again. Hence, I believe the second metric is more aligned with getting a churned user to return to Uber Eats as an active user. Suppose that a churned user returns with either an email reminder or discount within the first week. Then, if the same user returns the second week without any incentive, that’s an indication that the user is now active again.

[Interviewer] Great, now walkthrough the general procedure on A/B testing.

[Candidate] With the hypothesis statements and metrics defined, would determine the sample size for each group (control, treatment 1, treatment 2) based on practical signifiance (~ lift 1%), statistical significance level (0.05), power (0.8) and statistical test (pairwise T-Tests comparing proportions). The randomization unit will be conducted on churned users. Enough traffic should be allocated to achieve the desired power (0.8) within 1 to 2 weeks. Then, upon completion, conduct internal and external validity checks. If the experimentation ran soundly, I would evaluate the result to see if statistical significance has been reached.

[Interviewer] Okay, what would be your decision criteria based on the statistical result? *Try this question for 3 Minutes

[Candidate] Here’s my decision logic on how I would evaluate the result.

[While explaining, the candidate types notes on the Coderpad]

#Coderpad
No statistical significance on any pairs -> Consider changing the incentive and messaging first. Or, test different channel.
Both H1 and H2 are different from Ho, but they are the same -> If the effects of H1 and H2 are the same, then giving incentive is unnecessary and reduces Uber’s commission. Use H1 on churned users.
Both H1 and H2 are different from Ho, but H2 is greater than H1 -> Use H2 on churned users.

[Interviewer] Okay great, now one last question. Suppose that you went with H2, but you want to optimize on the discount %, main message and title. How would you test this? *Try this question for 5 Minutes

[Candidate] I would use a multivariate test. Given that there are multiple variations, it would make sense to choose a baseline first (the H2 version tested in the previous experiment), then compare the performance against other variations.

[While explaining, the candidate types notes on the Coderpad]

#Coderpad
Discount %’s: 15%, 20%Message Types: A, BTitle: A, B

For instance, suppose that there are three factors of interest with 3 levels in the discount %, 2 levels for the message type and 2 levels for the title. There would be 2 x 2 x 2 = 6 groups (1 control + 5 variations). Churned users would be randomly assigned to one of 6 groups. Then, the statistical result will be evaluated based on which variation resulted in the highest lift compared to the baseline against the baseline with statistical significance. So, instead of comparing every pairs which is unnecessary, just compare the lift % of the variation against the baseline. And, given that multiple tests increase the false positive rate, use a correction such as Bonferroni and Tukey HSD.

Interviewer Assessment

The candidate is assessed based on three criteria: business/product sense, statistical methodology and communication. Each dimension contains the following rating: (1) not competent, (2) marginal, (3) adequate, (4) good, (5) superior.

Business and Product Senses – 5

The candidate demonstrated strong business and product senses. He understands the main motive for incentivizing churned users is to get them to be active again. With that understanding, he chose an appropriate metric which measures the % of users who ordered the second time without an incentive. He also succeeded in not overlooking the definition of churn. Most other candidates would proceed with the analysis without first defining the meaning of a “churn” user. In contrast, the candidate ensures that “churn” are clearly defined, then assesses the trade-off in how it is defined. The candidate chose an appropriate churn definition given Uber Eat’s business model.

Statistical Methodology – 5

The candidate understands the procedure of running an A/B testing. He described the correct sequence from determining the sample size, randomization, experimentation time given traffic allocation to internal/external validity checks. Also, the candidate has demonstrated interpret an A/B test result by providing a business logic (i.e. If both H1 and H2 are different from Ho, but they are the same … use H1 to rollout). Lastly, the candidate understands how to run and analyze a multivariate testing.

Communication – 5

The candidate was clear and structured with his solution from the beginning to the end. he followed a logical sequence in explaining steps to solving the problem. Given that he didn’t seem to get stuck when follow-up questions were asked, he has demonstrated that he can think on his feet and has experience in running online testing.

[Uber] Incentivizing Inactive Users on Uber Eats

Problem

Hint

Unlock now by joining.