[LinkedIn] Engagement Increased, but Revenue Decreased

Problem

An experiment on LinkedIn Feed showed that the new ranking algorithm increases user engagement but decreases revenue. Should this be launched?

Tips

To address this product case question, structure the response in the following manner:

Business and Product Contexts – What does the interviewer mean by a “new ranking algorithm?” What is LinkedIn’s Feed? What are the UI/UX features of this product? What are the statistical outcomes of the experiment?
Metrics – What are the metrics that define “engagement” and “revenue “on Feed? What is the primary and secondary metric?
Analysis – Based on an initial set of details gathered from the interviewer, how would you interpret the outcome?
Recommendation – What is your recommendation to stakeholders?

Solution

[Interviewer] An experiment on LinkedIn Feed showed that the new ranking algorithm increases user engagement but decreases revenue. Should this be launched?

[Candidate] Thank you for the question. Before I propose a recommendation, I would like to clarify a couple of things. Do you mind if I ask you questions for clarification?

[Interviewer] Certainly.

[Candidate] First of all, you mentioned that the property in the Feed that was experimented with was the ranking algorithm. Is this ranking algorithm ordering how posts get displayed on users’ feeds?

[Interviewer] Yes, that’s correct.

[Candidate] Also, may I ask what metrics you use to track engagement and revenue?

[Interviewer] You will need to figure that out. Nonetheless, to give you a hint, the metrics should focus on Feed.

[Candidate] Gotcha. It’s my understanding that the Feed contains various media forms including texts, images, videos, and et cetera. I believe that the metric for engagement is the average minutes per user. This metric embodies the consumption of various media by a user.

Now, about the second metric that tracks revenue, does this revenue come from advertisements that are displayed on Feed or other products including premium subscriptions?

[Interviewer] That’s a good question. We only care about the Feed so it’s the revenue generated solely from the Feed.

[Candidate] Gotcha. The primary source of revenue generated from the Feed is advertisements. An appropriate metric will be the average advertisement revenue per user.

[Interviewer] Great. Any other questions?

[Candidate] Yes, I have one last question. Did the experiment show statistical significance in the directions?

[Interviewer] Good question. Yes, the results showed statistical significance.

[Candidate] Awesome. Thank you for the clarification. Do you mind if I take a minute to think about a recommendation?

[Interviewer] Certainly.

Commentary: Notice the way the candidate framed the problem. She made sure that the open-ended prompt was clearly defined before proposing a solution. She asked great questions about the algorithm, metric and statistical results. Once she clearly defined the problem, this allowed her to synthesize an approach. When you asked an open-ended question, do not jump into the solution right away. Take your time to understand the problem clearly. The interviewer is assessing your comprehension of the problem.

[Candidate] Based on the information I’ve gathered, I understand that this is an experiment that involves monitoring two key metrics. Typically, when there are multiple metrics of interest, one of them is primary while all the other ones are secondary. Can I assume that the primary metric is engagement while the secondary is revenue? Understanding this will help with the decisioning.

[Interviewer] Can you elaborate on that?

[Candidate] Typically an online platform such as LinkedIn that gains significant revenue from advertisement needs to balance post types: organic posts vs advertisement posts. If the user gets bombarded with advertisements, in the short term, the revenue may increase. But, in the long-term, user engagement may decrease. So, I believe that the primary metric is engagement while the secondary is revenue.

[Interviewer] Gotcha. Anything else?

[Candidate] Also, I think we need to create a decision matrix that clearly defines whether a feature should be ramped up or launched given the various outcomes. There is a total of 18 possible combinations of outcomes given that there are 3 directions (negative, positive, neutral) from the primary metric X 3 directions (negative, positive, neutral) from the secondary metric X 2 outcomes of statistical significance ( < 0.05 or >= 0.05 ).

I don’t believe we would have time to cover all aspects of this, but, in general, I believe that the product should be launched when the primary metric moves in the positive direction with statistical significance as long as the secondary metric is not negative and statistically significant.

In this problem case, we see that the primary metric moved in the positive direction at the expense of the revenue. Hence, my recommendation is that we do not proceed with this as this would chip away the revenue.

[Interviewer] Okay, what would you propose instead?

[Candidate] Unless we find any internal or external validities, I propose that we find a replacement for an algorithm that does not compromise the revenue as the engagement increases on Feed.

Interviewer Assessment

The candidate is assessed based on three criteria: business/product sense, statistical methodology, and communication. Each dimension contains the following rating: (1) not competent, (2) marginal, (3) adequate, (4) good, (5) superior.

Business and Product Senses – 5

The candidate receives a good remark based on the following instances:

She understood the product purpose and design of LinkedIn’s Feed.
She clearly defined metrics that serve as proxies for “engagement” and “revenue.” The metrics she defined, the average watch time per user (engagement) and average advertisement revenuer per user (revenue) were sensible.

Statistical Methodology – 5

The candidate demonstrated two instances of strength in his statistical methodology.

She demonstrated that experimentation results should be assessed with caution given that a product decision should be made only if internal and external validities are not compromised.
She understood how product decisions are based on the statistical outcome of an experiment.

Communication – 5

The candidate receives a superior remark based on the following:

Her responses were structured: (1) business and product contexts, (2) analysis, (3) recommendation.
She asked clarifying questions to frame the problem.
She provided a recommendation that was clear and precise.