Problem

How would you measure success on Instagram Stories?

Tips

Frame the open-ended problem in the following format:

Ask clarifying questions – (1) Confirm that your understanding of how Instagram Stories work is valid. (2) Who are the key stakeholders of the metric?
Metrics – Brainstorm and list potential metrics to track success on Instagram Stories.
Recommendation – How would you prioritize which metric is the most important?

Solution

[Interviewer] How would you measure success on Instagram Stories?

[Candidate] To address this question, I have a few follow-up questions. Stories is a short-form video that plays at most for 15 seconds. Additionally, viewers can swipe through one video to the next in an endless scroll. Is my understanding of the product, correct?

Commentary: Notice that the candidate has conducted her research on Instagram products before the interviewer. She most likely is an active user, or recently downloaded the application to understand the core features and products on the platform. When interviewing for a product-based role, it is vital to research the flagship product.

Ask yourself the following questions when deconstructing the application:

What is the main purpose of the application?
How does this application provide value to users?
How many active users are there?
What is the monetization strategy?
What are the UI and UX interactions that occur on the application?
What kind of data would be collected in this application?

Once you gathered key information about the application, this sets you up for success when asked about product-based interview questions.

[Interviewer] That is correct. Any other questions?

[Candidate] Yes, I do have one more. Who is the end-user of the success metric? I would imagine that we need to build a dashboard that tracks the success of this metric for a key audience. Is the audience composed of business stakeholders and product managers?

Commentary: Note that the candidate did not blindly list metrics. Every metric has a purpose and an audience in mind. The candidate started with the audience first, which allows her to tailor metrics for the stakeholders.

[Interviewer] Yes, that’s sound. Do you want to start proposing metrics?

[Candidate] Sure. I can list a number of metrics including the following:

# of reactions (i.e. emojis and likes)
# of shares
# of videos watched
# of hours watched
# of videos uploaded
Revenue generated from advertisement posts

Metrics 1 through 4 pertain to viewers, metric 5 pertains to influencers, and metric 6 pertains to monetization.

[Interviewer] Okay, anything else?

[Candidate] Formulating a metric needs to be specific. What I just listed tracks the key behaviors, but they don’t convey statistical functions (i.e. total, average, median) and unit of analysis (i.e. per user, per month, per week). So, I’m going to refine the metrics.

# of reactions per month
# of shares per month
# of videos watched per month
# of hours watched per month
# of videos uploaded per month
Total $ of advertisement revenue per month

[Interviewer] You listed per month, why did you pick this time period? Why not one day or one week?

[Candidate] This is because the end-user is a business stakeholder who will be viewing the metric over a long horizon. They typically make decisions based on monthly historical data, rather than daily or weekly datapoint. For instance, to gauge the business growth, they would be interested in how the $ revenue advertisement grew YoY in April.

[Interviewer] Okay, what about annual?

[Candidate] We can certainly provide an annual metric as well. Really, depending on the use case, the metric can be reformed monthly or annually. It really boils down to how the stakeholders themselves will use the metrics to report and make decisions.

[Interviewer] Gotcha, what about per-user? Would you want to look at the average # of videos watched per user or the total # of videos watched across users?

[Candidate] The latter; the first metric is useful in AB testing. However, for stakeholders who want to understand the growth of the platform as a whole, they would want to look at the total # of videos watched as it accommodates for both increased engagement of existing users and onboarding of new users.

[Interviewer] Now, if you had to prioritize three metrics, which ones would you select? And, among those three, how would you rank order the metrics?

[Candidate] I would say that the following should be kept: (1) Watch time in hours per month, (2) Total # of videos published per month, (3) Advertisement $ per month. The first metric embodies all the interactions (i.e. watch, swipes and reactions) of the Stories.

It embodies the active engagement on the application and highly correlated with advertisement dollars. So, in fact, this should be ranked as #1.

The second metric pertains to the influencers on the platform. We want to see that they are producing content that promotes user engagements. So, this should be ranked as #2.

The third metric is monetization as it serves the bottomline of Instagram’s business model. As the engagement grows, so should the revenue. For business stakeholders, this is one of the key metrics they care about. It should be ranked as #3, but not before the key engagements metrics. This is because we want to avoid overly optimizing the revenue that is inorganic to users. If advertisement $ is the most important metric, then users will get spammed with ads and churn in the long term.

Interviewer Assessment

The candidate is assessed based on three criteria: business/product sense, statistical methodology and communication. Each dimension contains the following rating: (1) not competent, (2) marginal, (3) adequate, (4) good, (5) superior.

Business and Product Senses – 5

The candidate has conducted her research on Instagram products. She demonstrated her understanding of a core feature on Instagram, which is Stories.
The candidate understands that metrics have an audience in mind. She ensured that she understands who are the end-users of the success metric before designing it.

Statistical Methodology – 5

The candidate provided a comprehensive list of metrics to consider before choosing a primary metric that the business should align on.
The candidate understands the accounting of a metric: key behavior and unit of analysis.

Communication – 5

The candidate’s response was structured: (1) clarifying questions, (2) brainstorming, (3) recommendation.
She asked clarifying questions to understand the problem before proposing metrics.
She explained her points clearly and comprehensively.

[Instagram] Measure Success on Stories

Problem

Tips

Unlock now by joining.