Problem

How would you measure active users on Instacart?

Hint

This is an open-ended question that requires you to deconstruct the problem and provide a structured response. Here’s an outline on how to address a question that asks you to measure active users of a product:

Understand the Business Model
- What’s the business model? How is it helping customers in exchange for their business?
Define the Business Purpose
- How will the metric be used?
- Who is the intended audience of the metric?
Define the Main Action
- What’s the key behavior that defines an active user?
Define the Accounting of an Active User
- What is the formula for measuring active users?
Explain How Active Users Will be Monitored
- How will the metrics be monitored for stakeholders?

Solution

[Interviewer] How do you measure active users at Instacart?

[Candidate] Thank you for the question. To address this question, I’d like to first echo my understand on the business model of Instacart and ask a few claryfing questions. I think it’s important to explain the business context and get clarity before offering a technical solution.

Commentary: Note how the candidate began. Instead of claiming, “the metric for active users in Instacart is X,” she is first scoping out the business context and assumptions. An interviewer will see this as a plus in terms of business/product sense and communication.

[Interviewer] That sounds good.

[Candidate] Based on prior research, I understand the Instacart is a multi-sided marketplace that serves as a coordinator among grocery customers, shoppers (contractors) and stores (i.e. Costco). Instacart generates revenues from orders and delivery fees. Now, I’d like to ask if a few clarifying questions. When you asked to measure active “users,” which type of users are you referring to? Is it the customers or shoppers?

[Interviewer] That’s a good question. In this case, I’d like to understand how active users are measured on the customer-side.

[Candidate] Understood, since it’s the active customers we care about, that makes sense. Obviously, they are perhaps the most important revenue drivers of the Instacart’s business model. Because, if you do not have customers, you have no orders, which means no revenue for Instacart. Now, I need to identify one key action that defines an “active” user. It’s usually based on a behavior that is the most correlated and meaningful to revenue. In this case, it will be orders. So, we can track “active” users on Instacart by assessing their order volume.

Commentary: The candidate has provided the next logical step to her explanation. She discussed a primary action that serves as a proxy for “active” user. Every products, whether it’s Facebook, Google Search or Netflix, have primary action that’s most reflective of an active user and correlated with the bottom line of a company’s financials. Identifying this key action is crucial to demonstrating that she has product-sense.

Here’s a pro-tip: name a few apps you often use and identify a primary behavior that would label you as an “active” user.

[Interviewer] Okay, that sounds good. What’s the next step?

[Candidate] Now, that we have identified the key user action to track. The next logical step is to determine the accounting of active users.

The general formula for active user is the following:

[While explaining, the candidate types notes on the Coderpad]

#Coderpad

Active User(t) = New User(t) + Retained User(t) – Churned User(t)

For some fixed window t, we need to tally how many of the users have ordered for the first time, repeated ordering and stopped ordering.

We also need to identify an appropriate window for binning a user into one of the three buckets. The standard forms are daily (DAUs), weekly (WAUs) and monthly (MAUs). We would want to assess all of them as provide slightly different information.

[Interviewer] Interesting, can you elaborate on DAUs vs WAUs vs MAUs? And, if you had to choose one of those three, which one would you choose?

[Candidate] Sure thing. Generally, the time window is chosen based on the frequency of the action. This can be informed based on business intuition/survey and data. For instance, a customer typically shops once a week, and this assumption can be backed by data. We can take a look at the poisson distribution of order frequency at an interval of 30 days (~ 1 month). The mean of the poisson distribution informs how often a customer orders every 30 days. Now, this mean can be reciprocated such that we now have the mean of the exponential distribution, which informs the average time interval in days between orders.

[While explaining, the candidate types notes on the Coderpad]

#Coderpad

Posson Mean => Exponential Mean => Interpretation

1 order / 30 days => 30 => User orders once a month (1 month)

4-5 order / 30 days => ~7.5 => User orders every 7.5 days (~ 1 week)

30 order / 30 days => 1 => User daily

Interviewer Assessment

The candidate is assessed based on three criteria: business/product sense, statistical methodology and communication. Each dimension contains the following rating: (1) not competent, (2) marginal, (3) adequate, (4) good, (5) superior.

Business and Product Senses – 4

The candidate received a good remark based on the following instances:

She understood the business model of Instacart.
She was spot on with a key user action that serves as a proxy for “active.”

One suggestion that could have helped her demonstrate a superior response.

The candidate should have provided details on how the business purpose of measuring active users and the metric will be monitored (i.e. dashboard) for stakeholders.

Statistical Methodology – 5

The candidate demonstrated two instances of strength in her statistical methodology.

The candidate demonstrated that she understand the accounting formula for tracking active users: Active User(t) = New User(t) + Retained User(t) – Churned User(t). This is a sound approach.
The candidate was familiar with statistical distributions of a customer behavior in ordering.

Communication – 5

The candidate received a superior remark based the following:

Her responses were structured (buinses context => identify key user action => formula) and easy to follow.
She ensured that she clarifies her assumptions before approaching the problem.
She used the coderpad to jot down ideas. She reinforced her key message which was easy to follow.

[Instacart] Measure Active Users

Problem

Hint

Unlock now by joining.