Problem

Duration: 20 Minutes
Difficulty: Medium

An interviewer at Facebook asked:

If you want to increase the % of posts in Facebook Groups that receive a comment, how would you do this?

Hint

The interviewer is looking to see if you can produce a structured response to an open-ended business problem. Rambling without a structured response will cost you points in a case interview. Use the following framework to address the interview question:

Clarifying Questions
- What are the questions you would ask the interviewer to understand the problem?
- What are your initial assumptions that you need to clarify?

Business and Product Contexts
- What is the company’s mission statement (OKR)?
- What is the product goal and features? How does it relate to the company’s mission statement?
- What is the intention of the current product objective?
- What is the KPI of interest?

Business Hypothesis
- What are your initial assumptions on features that can drive the KPI?
- What are your business hypothesis questions?

Analysis Data and Methodology
- What kind of data source and variables would you need to address the business hypothesis question?
- What is your analysis design? What does your target variable look like?
- What is your statistical methodology? Specify the name of a statistical model or test. How would you use it?

Decisioning and Recommendations
- How would you interpret the results from your methodology?
- What is your recommendation to the business/data/product leads?
- What does success look like to you?

Candidate Example

[Interviewer] If you want to increase the % of posts in Facebook Groups that receive a comment, how would you do this? *Try this question for 10 Minutes

[Candidate] Thank you for the problem. I’d like to first confirm my understanding of the feature. Facebook Group is a forum that allows Facebook users to create, join and share posts, right? Are there any particular metrics that the business may care about?

Commentary:
Notice that the candidate begins with questions to clarify. In an open-ended case problem, it’s vital to ask for three practical reasons: (1) It helps brainstorm ideas, (2) constrains the open-ended problem in terms of what to solve or avoid, (3) shows that you are an active listener which is an important communication skill to convey to the interviewer.

[Interviewer] Yes, you are correct about the Facebook Group. In terms of the metric, you will need to guide the business.

[Candidate] Thank you for the clarification. To address the problem, I think it’s useful to briefly discuss the business context.

[While explaining, the candidate types notes on the Coderpad]

#Coderpad
Business Contexts——————————
Mission Statement: To build a community and bring the world closer togetherBusiness Goal: Improve engagement (i.e. comments) on Facebook Group Posts Potential Impacts: Increases time spent on Facebook, which in turn increases advertisement revenue.

I understand that Facebook’s mission statement is to build a community and bring the world closer together. Clearly, what Facebook group wants to achieve is improve engagements on Facebook Groups. Increasing comments on group posts could mean that Facebook users are spending more time on the platform, which in turn grows advertisement revenue.

Commentary:
Take notes on what the candidate did superbly. She demonstrated “business-sense” with her discussion of the business contexts on the product problem. The interviewer will see this as a plus. This also provides a background for her to ideate potential solutions to the problem. Secondly, she illustrated her points on coderpad. This is a plus that many candidates don’t consider when responding to a verbal question. The interviewer can see that her communication skill is strong.

[Interviewer] Those are all great points, what’s the next step? *Try this question for 5 Minutes

[Candidate] I think the next step is to define a KPI that measures the business objective: increase the % of posts that receive a comment in Facebook Groups. To define the KPI, I would like to brainstorm options.

[While explaining, the candidate types notes on the Coderpad]

#Coderpad
Metric———–
% of Posts that Receive at least 1 Comment Average # of Comments per Posts

First, I consider a metric that measures the “% of Posts that Receive at least 1 Comment.” I believe this best measures the business objective. Another metric I consider is the “Average # of Comments per Posts.” This conveys the density of post engagements via comments. The second metric, however, is less precise than the first one in measuring the main objective. So, I would use the first one as the business KPI to optimize.

Commentary:
Take a quick pause to think about what the candidate did right and/or wrong.
This would be the interviewer’s take:
Pros:
The candidate understands what is the next logical step in solving the problem.The candidate’s thought process is clear as illustrated on CoderPad.
Cons:
Though the candidate has the right ideas on metrics, still they need to be refined. Both metrics lack the units of analysis. “% of Posts that Receive at least 1 Comment” with respect to what time? Per day, Per week, Per month? Also, when will the candidate count success? A post receives its comment 30 days later, should this be counted or disregarded given the latency in the comment response time?
When creating a metric, consider two main ingredients:
Measures of Behavior – Actions that define success. A post that receives a comment within 24 hours could be considered as a success, but 30 days later may not be. A cut-off needs to be clearly defined on what is a “success” action. This is not only relevant for user-experience, assuming that users seek responses sooner than later, but also the business cannot wait that long to tally the successes.
Unit of Analysis – What is the duration of the action observed? Per day? Per week? Based on the primary measure that the candidate proposed “# of posts that receive at least 1 comment,” it’s implied that the calculation is the lifetime value since Facebook first established Groups. Such a metric isn’t helpful. The candidate should refine in terms of a window to track performance over time.

[Interviewer] The metrics seem incomplete as they seem to compute the lifetime values. However, what if you want to track how the metrics change over time? Also, how would you account for instances when posts take a long time to receive a comment? *Try this question for 5 Minutes

[Candidate] Ah, yes, let me address your questions. First, I can consider the observation window as per day, week or month. In this case, per day is too granular, and, depending on the day of the week or month, the proportion of posts with success is prone to variability due to seasonality. Hence, per week or month is reasonable. I would use per-week in an experimentation, but, when reporting it to an executive, provide the monthly %. Essentially, the metric would be “% of Posts that Receive at least 1 Comment per week/month.”

To address your second question, obviously, we can’t wait days to see if a post receives a comment. So, a cutoff on the time delay is required to tally success. To find an idea cutoff, I can look at the cumulative distribution of the hours it takes for a post to receive its first comment. Then, I can find a cutoff point based on a percentile, let’s say 90th percentile. Let’s presume that the 90th percentile represents 24 hours. Hence, a post achieves success based on the following criteria:

[While explaining, the candidate types notes on the Coderpad]

#Coderpad
Success————–
Let t be the delta (hours) between a post timestamp and the first comment timestamp.
Success = 1, t <= 24 0, t > 24

If a post receives a comment within 24 hours, then the event is a success. Otherwise, it’s a fail.

Commentary:
Take a quick pause to think about what the candidate did right and/or wrong.
This would be the interviewer’s take:
Pros:
The candidate understands what is the next logical step in solving the problem.The candidate’s thought process is clear as illustrated on CoderPad.
Cons:
Though the candidate has the right ideas on metrics, still they need to be refined. Both metrics lack the units of analysis. “% of Posts that Receive at least 1 Comment” with respect to what time? Per day, Per week, Per month? Also, when will the candidate count success? A post receives its comment 30 days later, should this be counted or disregarded given the latency in the comment response time?
When creating a metric, consider two main ingredients:
Measures of Behavior – Actions that define success. A post that receives a comment within 24 hours could be considered as a success, but 30 days later may not be. A cut-off needs to be clearly defined on what is a “success” action. This is not only relevant for user-experience, assuming that users seek responses sooner than later, but also the business cannot wait that long to tally the successes.
Unit of Analysis – What is the duration of the action observed? Per day? Per week? Based on the primary measure that the candidate proposed “# of posts that receive at least 1 comment,” it’s implied that the calculation is the lifetime value since Facebook first established Groups. Such a metric isn’t helpful. The candidate should refine in terms of a window to track performance over time.

[Interviewer] Why don’t you look at the average # of comments per post, instead? *Try this question for 5 Minutes

[Candidate] Well, although the average # of comments is an important metric, it does not address coverage of posts that receive comments that the business seeks. The average comments may advance; however, that does not mean that the coverage will move. Hence, the direct target to focus on is the “% of posts that receive a comment, successfully.”

Commentary:
Oftentimes, candidates get stomped if an interviewer appears to repudiate an approach. Such a question is not intended to correct your approach. Rather, the interviewer is assessing the candidate’s ability to think on her feet and defend her method. In other words, there is no definitively right or wrong answer. Rather, the candidate is assessed based on her ability to share an approach with a sound reasoning.
In this case, the candidate pointed out a flaw in using the average, then supported the reason for using the % of posts that receive comments as the target metric to analyze.

[Interviewer] Okay, that’s good. Now, how would you run your analysis? *Try this question for 5 Minutes

[Candidate] To conduct an analysis, I will need to formulate hypothesis questions, acquire data, run EDA, build a statistical model and analyze results. I would like to first discuss hypothesis questions. These will help me identify potential factors that can increase coverage.

[While explaining, the candidate types notes on the Coderpad]

#Coderpad
Guiding Questions——————————
As friend views increase, what is the likelihood of a post receiving a comment? Does the group membership size have an impact on the likelihood of a post receiving a comment?Are there devices/browsers that seem to comment less?

The first one shows whether there’s a correlation between friend views and the likelihood of a post receiving a comment. If the relationship is positive with statistical significance, this is perhaps an opportunity to improve using A/B testing.

The second one shows whether a group with larger membership will increase comment coverage than the smaller ones. I would imagine that posts in large groups are more likely to receive comments given that more people view them. If that’s the case, then there should be an additional investigation to facilitate growth in groups.

The third one reveals which device appears to lag when it comes to post engagement. For instance, if Firefox users on Windows are less likely to comment, perhaps there are bugs or poor UI designs that discourage the cohort from reacting to posts.

[Interviewer] Sounds good. Now, what’s the next step? *Try this question for 5 Minutes

[Candidate] The next step is to find relevant data to conduct the analysis. I would imagine that Facebook would gather the following data:

[While explaining, the candidate types notes on the Coderpad]

#Coderpad
Data Sources———————-
User profile data (i.e. age, gender, geo)Group data (i.e. membership count, category)Connections data (i.e. friend connections)Device (i.e. MacBrowsers (i.e. IE, Firefox, Chrome)Geo (i.e. Country, Region)Post (i.e. Publication Timestamp, Character Length, Text)

Commentary:
Pro tip to prepare for a company interview. Preemptively list types of data company/team X would gather for their data science problems. When you enter an interview, you will be more prepared to address case questions. Run through a similar exercise in listing possible data sources that other companies such as Apple, Netflix and Amazon might collect.

[Interviewer] What’s the next step? *Try this question for 5 Minutes

[Candidate] The next step is to prepare the data for a statistical model. Using the data source above, I can create the following model features:

[While explaining, the candidate types notes on the Coderpad]

#Coderpad
Features————–
Post Timestamp# of Views # of Friends Viewed Group SizeGroup Category# of IE Viewers# of Firefox Viewers…..U.S. IndicatorLanguage: English
Target Variable ———————–First Comment [0, 1]

Note that each feature value is a moving variable. So, the # of views at publication vs success/failure point will change. Hence, what we care about is the snapshot when the success/failure occurred. Also, I’ve only displayed ten as an illustration. If I were to conduct an actual analysis, I would probably identify hundreds then apply feature selection using regularization to extract key features.

[Interviewer] Great, now what kind of model would you use? *Try this question for 5 Minutes

[Candidate] Given that this is an analytic exercise, I will choose a model with high interpretability. Such a model can be the logistic regression or random forest with the partial dependence plot.

[Interviewer] Now, let’s wrap this up, how would you provide a recommendation to the business team? *Try this question for 10 Minutes

[Candidate] The recommendation will be largely dependent on the result from the statistical analysis. Let’s presume that the logistic regression model was used, and there is statistical significance on the positive relationship between # of friends’ views and the likelihood of a post receiving a comment.

[While explaining, the candidate types notes on the Coderpad]

#Coderpad
Model Results—————-
# Logistic Regression
Log(Odds) ~ Intercept + B * Friend_Views + …. + B.k * X.k

Let’s also presume that the beta coefficient for “friend_views” is >0 in the logit form, which means that the odds ratio is greater than 1. That means that, holding all other variables constant, for every 1 unit of increase of friend_views, the odds of success (the first comment within 24 hours) increases by [exp{B * Friend_Views} – 1]%. For every 1 unit of increase in friend views, I can present a table of odds, percentages and expected values.

#Coderpad
Model Results—————-
# Summary
*The values are made-up for this exercise*Expected Success Count = % * 1M Group Posts
———————————————————————————————————————————–| # of Friend Views | Expected Odd | Expected % | Expected Success Count |———————————————————————————————————————————– 0 0.8 44% 440,000 (Base) 1 0.9 47% 470,000 (+30,000 from Base) 2 1.013 50% 500,000 (+60,000 from Base)

I would share the table summary to the business, and show that, if the friend views increase from none to 1, they can expect to increase coverage from 44% to 47%. In other words, for every 1 million posts published, the incremental increase in posts receiving the first comments is 30,000.

This will provide a justification for a follow-up in the form of A/B testing. I can recommend to the business that they consider UI/UX changes that could increase the exposure of a group post among friends. Perhaps, a group post could appear on a friend’s feed even though he or she isn’t enrolled in the group. If they like the idea, then I can guide them on A/B testing to validate the idea.

Commentary:
A naive response would just convey that there is statistical significance on the positive relationship between friend_views and the likelihood of comments; thereby, the business should run an A/B testing to encourage friend_views.
The candidate provided concrete details from the analysis to the final recommendations. She shared how she would use and interpret the statistical model and translate the results into numerical summary that the business can understand. This sets the stage for her to provide a recommendation.

Interviewer Assessment

The candidate is assessed based on three criteria: business/product sense, statistical methodology and communication. Each dimension contains the following rating: (1) not competent, (2) marginal, (3) adequate, (4) good, (5) superior.

Business and Product Senses – 5

The candidate receives a superior remark given that she laid the business context before diving into solution. She mentioned Facebook’s mission statement and how it potentially ties into the problem. This suggests that, if she were a data scientist in Facebook, she won’t just blindly accept a project to solve. Rather, she will understand the business context, then formulate a data science approach to solve it. She also seems to understand how to communicate her analysis message in a way that the business can understand. Unlike most other candidates who would end their analysis with “run an A/B testing,” she, on the other hand, stood out by providing justification for her recommendation to run an A/B testing.

Statistical Methodology – 5

The candidate has demonstrated superior skills in her statistical methodology. First, she was clear with the overall structure of how to run a statistical analysis. She discussed that she would formulate hypotheses, gather data, prepare model data, run a statistical model, and conduct an analysis to provide a recommendation. For each of the analysis steps, she was comprehensive and precise with her approach. For instance, most candidates would overlook the details involved in creating the label. A naive response would say that, “if the post receives a comment, then tally that as success.” In contrast, the candidate understands that there are additional considerations for creating the label. She defined a constraint: “if a post receives a comment within 24 hours, then the event is a success. Otherwise, it’s a fail.”

She also provided detailed responses on the model data. A naive response will just say: “just use data to run a statistical model.” In contrast, she showed what kind of data sources and features she would use to address her hypotheses.

Lastly, her choice of statistical model (i.e. the logistic regression model) and interpretation were solid. She demonstrated that she has a quality technical ability in statistics.

Communication – 5

The candidate was very structured with her solution from the beginning to the end. She followed a logical sequence in explaining steps to solving the problem. First, she started by asking questions to clarify and explaining business contexts. This is a crucial step in almost any data science project. Next, she then fleshed out the target metric to optimize. Then, she proceeded explaining the statistical procedure to solve the problem. Lastly, at the end of the analysis, she tied her statistical findings and interpreted the result in a way for the business to understand.

[Meta] Facebook Groups

Problem

Hint

Unlock now by joining.