Problem

How would you measure the performance of the News Feed on Facebook?

Hint

This is an open-ended question that requires you to deconstruct the problem and provide a structured response. Here’s an outline on how to address a question that asks you to measure a key product within a platform:

Product Awareness
- What is the news feed? How does it work?
Business Purpose
- What is the intention behind measuring the “performance” of the news feed?
- Who is the intended audience of the metrics?
Key Metrics
- What are the key metrics? What are their purpose?
- If you had to choose one primary metric, what would it be?
Monitoring
- How will the metrics be monitored?

Solution

[Interviewer] How would you measure the performance of News Feed on Facebook?

[Candidate] Thank you for the question. Before I start brainstorming metrics to evaluate the performance of the news feed, I’d like to first explain what it is.

I know that the news feed is one of the core features of Facebook that curates an endless scroll of posts published by friends, groups and advertisers. This enables users to seamlessly stay up-to-date on the latest events happening across the user’s network of social connections. At the same time, it serves as one of Facebook’s core real estate to display advertisements to grow Facebook’s monetization.

Commentary: The candidate’s entry to the question is strategic. A naive response will open with “Here’s a list of metrics I would use: X, Y, Z.” The candidate understands that context matters before proposing a solution. Hence, he is establishing the product function first before proposing metrics. It’s important to demonstrate that you understand what you are measuring before you discuss how you will measure it.

[Interviewer] That sounds good.

[Candidate] Now, may I ask who is the intended audience for the metrics? Is it for the executives and/or relevance team in Facebook?

[Interviewer] Good question, let’s presume that we want a metric or a set of metrics that defines whether the feed is performing well or not.

[Candidate] I see. It seems to me that we need to define a metric or a set of metrics that serves as a proxy on the “growth” of the news feed. This will essentially serve as the north star metric that all product decisions (~ A/B testing, recommender system, UI/UX changes) will try to optimize on. It will also serve as a metric that executives will socialize to track whether the core feature is growing.

Commentary: Notice the way the candidate scoped the purpose of measuring the news feed the first place. Every metric has an underlying business or technical context. Make sure to call it out before listing metrics as this demonstrates both product and business sense to the interviewer.

[Interviewer] Great, can you define metrics you would use to evaluate the performance?

[Candidate] Sure thing. Typically in measuring a product, there are two sets of metrics: (1) usage and (2) revenue. Now, considering that the feed provides ranks posts to read based on relevance, in this particular example, there should be a third set (3) relevance.

[While explaining, the candidate types notes on the Coderpad]

#Coderpad

Metric Categories

Usage: Any user engagement which includes posts, reads, comments, reactions, shares and clicks.

Revenue: Any revenue generated from advertiser posts.

Relevance: Technical measures of whether the feed is finding and ranking relevant posts to users.

I would say that the first two are most relevant for executives and business analysts. The third is mostly relevant for data scientists and machine learning engineers who need evaluate the quality of the recommendations. Is the feed curating relevant posts and ranking the posts in the right order? Is the feed optimizing on both revenue and engagement without maximizing on one aspect over the other?

[Interviewer] Interesting, do you want to start listing a set of specific metrics to monitor?

[Candidate] Sure thing. Here are a couple I can think if we can be expressed in the form of sum, average, proportion and time windows (i.e. daily, weekly, monthly).

[While explaining, the candidate types notes on the Coderpad]

#Coderpad

### Usage

Total

# of Posts
# of Reads
# of Comments
# of Reactions
# of Shares
# of Clicks
# of Engagements (~ SUM(Posts, Reads, Comments, …., Clicks)
Time Spent on Feed

Average

Average # of Posts / User
Average # of Reads / User
Average # of Comments / User
Average # of Reactions / User
Average # of Shares / User
Average # of Clicks / User
Average # of Engagements / User
Average Time Spent (minutes) / User

### Revenue

Total

# of impressions
# of advertisement clicks
Total revenue generated from clicks

Average

Average impressions
Average CTR
Average % of Advertiser Posts per User Session (i.e. a user views 10 posts in the feed in a session, 2 of those posts are advertisements. Average the % across users)

### Relevance

Mean Average Precision (MAP)
Other variations of precision-rank based evaluations

All of the metrics outlined can be confined into some time window depending on the audience. For instance, we could look at the average # of engagement per user per day. The trend of this metric will show whether users are in general continuing to use the feature or letting go.

Commentary: The candidate provided a decent strategy to define the metrics of a product. Instead of providing a list with no coherent structure. Identify the logical groupings or themes of a metric. Then, provide a list within the grouping.

Pro Tip! Do keep in mind the science of metric. To create a metric, you need to consider two ingredients: Unit of Behavior and Unit of Analysis. The unit of behavior defines what is the behavior you are measuring and how you are counting it (~ Averaging, proportion, sum), and the unit of analysis is the dimension of how the counting occurs (per user-session, per session, per day, per week and e.t.c.)

[Interviewer] That’s a comprehensive list. If you had to choose three metrics, which one would you use?

[Candidate] I will first presume that the intended audience is for executives and product leads. Perhaps we want to display the metrics on a dashboard so trends can be tracked over time. This will help business stakeholders understand whether they are moving in the right direction to move the needle in terms of business and product strategies.

So, in this sense, I believe I believe we need a metric that provides value to the user. In this case, I believe it will be the total time in hours spent per day. This will accommodate for variations in post content as some text-based posts are short while others are long. Secondly, some posts are videos so if a user spends 20 minutes watching a video, the metric will count those as time spent on the feed. Thirdly, if a user clicks an external link, we won’t be tallying those instance as we just lost a user. So, the total time spent will serve as the north star metric to understand the success of the News Feed.

The second aspect is the revenue. The total $ revenue per month or quarter will be relevant. This should be a larger window as, usually, business accounting is performed at the monthly, quarterly and annual level. This will help us provide a high-level overview of whether Facebook is helping to advertisers target the right users and how it contributes to Facebook’s financials.

The third aspect is the average % of advertiser posts per user session. Facebook needs to find the sweet spot the optimize on advertiser revenue and organic posts. If the % is too high, let’s say 50%, that means for every 1 posts from a friend, the next post in the queue will be an advertisement which feels spammy. So, the user might churn. At the same time, if it’s too low, then Facebook won’t generate revenue. So, a sweet spot will need to be identified, and tracked overtime to ensure that there’s no volatility with the equilibrium despite various changes to the product over time.

Interviewer Assessment

The candidate is assessed based on three criteria: business/product sense, statistical methodology and communication. Each dimension contains the following rating: (1) not competent, (2) marginal, (3) adequate, (4) good, (5) superior.

Business and Product Senses – 5

The candidate receives a good remark based on the following instances:

He understood the product purpose and design of Facebook’s news feeds. He mentioned that Facebook’s news feed is an endless scroll that curates posts from friends, groups and advertisers. This shows that the candidate is familiar with Facebook’s flagship product.
He understood the intended users of the metric. Instead of merely listing metrics, he provided business and product contexts on why the metrics should be used.

Statistical Methodology – 5

The candidate demonstrated two instances of strength in his statistical methodology.

The candidate provided a comprehensive list of metrics, listing about two dozen metrics. He also seems to understand the science of defining a metric as he provided metrics that included unit of behavior and unit of analysis.

Communication – 5

The candidate receives a superior remark based on the following:

His responses were structured: (1) business context, (2) brainstorming, (3) recommendation.
He asked clarifying questions to frame the problem.
He illustrated points on the Coderpad to ensure that key messages are expressed.

[Meta] Measuring Performance of News Feed

Problem

Hint

Unlock now by joining.