Metrics & KPIs Interview Questions

Sample Answer

Use long click rate as the primary success metric, the share of queries where the user ends the session satisfied without immediate reformulation. It directly reflects whether the summary helps users complete the task, not just click more. Guardrail with reformulation rate within a short window, and with downstream dissatisfaction signals like pogo sticking or reduced dwell time on clicked results. If long clicks rise while reformulations and pogo sticking do not, you have evidence the summary improves outcome quality.

Sample Answer

You could optimize request to trip conversion, or you could optimize completed trips per eligible session with guardrails. Conversion wins here because it can increase from users confirming more but canceling later, which harms drivers and increases wasted supply. A stronger primary metric is completed trips per rider session, paired with guardrails like rider cancel rate, driver cancel rate, and time to match. This aligns to the real outcome, a finished ride, while explicitly protecting marketplace reliability.

Sample Answer

First, define what discovery means in this surface: more users starting something they would not have otherwise, not just more scrolling. Next, choose a primary metric like incremental qualified starts per member, where qualified means watch time exceeds a threshold such as $t \ge 10$ minutes or completion rate above a cutoff for episodic content. Then add satisfaction guardrails: short play and abandon rate, thumbs down rate, and next day retention. Finally, ensure the decision rule is explicit, ship if qualified starts rise and guardrails stay flat or improve, otherwise diagnose whether you are creating low quality sampling.

Sample Answer

You could use MAU or you could use "fulfilled orders from retained customers". Fulfilled orders wins here because it bakes in value delivery, it excludes failed experiences, and it links directly to unit economics. Your tree can be $\text{Orders} = \text{Active customers} \times \text{order frequency} \times \text{checkout conversion}$, with a fulfillment quality branch for on time rate, cancellation rate, and refunds. To prevent local optimization, you set guardrails, for example require cancellation rate and refund rate to not worsen while increasing orders.

Sample Answer

I would start from the outcome Airbnb exists to create, successful nights booked, then define it precisely as completed stays, not bookings created. Next I break it into demand and supply matching: nights booked equals searchers times booking conversion times average length of stay. Booking conversion then splits into availability coverage, price competitiveness, and product trust signals like review quality and host acceptance. Finally I add guardrails for trust, for example cancellations, rebooking friction, and customer support contacts, so growth does not come from low quality supply.

Sample Answer

First, define the north star outcome like completed trips per user over 28 days, and decompose it into request rate, match rate, cancel rate, and supply availability. Next, forecast each component using early post-change data, for example a hierarchical model by city to borrow strength where data is sparse. Then recombine: $$\Delta \text{Trips/User} \approx \Delta r\cdot m\cdot (1-c) + r\cdot \Delta m\cdot (1-c) - r\cdot m\cdot \Delta c$$, and propagate uncertainty to get risk bounds. Your early failure signals are match rate, driver online hours, and rider ETAs, if those do not improve quickly, the long-term offset is unlikely.

Sample Answer

This question is checking whether you can separate short-term engagement sugar highs from metrics that predict durable value and avoid Goodhart effects. You propose lagging outcomes like D30 retention, creator churn, and content diversity, plus leading signals like week-1 return frequency, hides, unfollows, negative feedback rate, and creator posting velocity. You set decision rules that require leading signals to be directionally consistent with the intended long-term outcome, and you pre-register guardrails, for example negative feedback cannot rise more than $x\%$. If watch time rises but negative feedback and creator supply worsen, you treat that as a high-risk pattern and pause or roll back despite the immediate gain.

Sample Answer

This question is checking whether you can anticipate incentive gaming and protect the system with measurable guardrails. If you pay on pickup ETA, drivers can camp in dense areas, reject longer pickups, or churn after quick pickups, which can worsen coverage, cancel rates, and marketplace balance. You propose either an incentive based on completed trips with service level constraints, or a balanced scorecard: pickup ETA plus acceptance rate, cancellation rate, and coverage in underserved zones. You also monitor rider outcomes like on time pickup and complaint rate, because ETA alone is easy to manipulate.

Sample Answer

The standard move is to optimize for long term value, so completion weighted viewing or retention should outrank raw starts. But here, starts can be good discovery, and completion can drop simply because users sample more, so you need a quality adjusted metric like hours watched per visitor capped per title, or completed episodes per start. Set a guardrail on early abandonment, for example % of plays abandoned in under 2 minutes, plus 7 day return rate. Decide with segment cuts, new users may benefit from sampling, while existing users might show fatigue.

Sample Answer

Get this wrong in production and you can shrink selection, hurt merchant revenue, and increase churn even if deliveries are faster. The right call is to guardrail marketplace health: conversion rate, basket size, reorder rate, merchant fill rate, dasher utilization, and cancellation and refund rates. You also track long tail exposure so you do not starve certain merchants, and you segment by dense versus suburban zones where the trade off differs. If you must optimize speed, use a constrained optimization, minimize delivery time subject to conversion and reorder not declining beyond a threshold.

Sample Answer

More send volume sounds reasonable but breaks under recipient fatigue, which lowers response rate and can drive member churn. Using send volume alone does not work because recruiters can blast generic messages and still hit the metric. That leaves outcome based incentives: replies per InMail, hires attributed, and a recipient quality score with guardrails on blocks, spam reports, and member session decline. You also cap sends per recruiter per day and require personalization signals, because constraints reduce the payoff to gaming.

Sample Answer

The standard move is to decompose $\text{CR} = \frac{\text{conversions}}{\text{eligible sessions}}$ and check which term changed, then slice by variant, platform, and app version. But here, denominator integrity matters because eligibility often changes with experiments, like redefining what counts as a checkout start, filtering failed sessions, or shifting traffic to a different flow. You validate invariants, like stable counts of upstream events and stable exposure assignment rates, and you confirm event schema and logging coverage did not change at launch time. Finally, you rerun the metric using a frozen denominator definition, for example eligibility based on a pre-experiment event, to test whether the lift persists.

Sample Answer

Get this wrong in production and you ship a rollback that kills real growth, or you celebrate a fake win while users churn from errors. The right call is to reconcile hours watched computed from independent signals, like player heartbeat events, CDN bytes, and session start stop boundaries, then look for Android-specific changes in event rates and duplicate events. You check whether retries or error loops inflate heartbeat counts, and whether sessionization rules changed, like extending sessions when the app is backgrounded. Then you validate against user-level distributions, like median minutes per user and fraction of users with extreme watch time, because artifacts often show up as long-tail blowups.

Sample Answer

Averaging the two numbers sounds reasonable but breaks under inconsistent denominators. Declaring one dashboard wrong without auditing definitions doesn't work because both can be internally consistent with different product questions. That leaves you to write down the canonical metric spec, event grain, join keys, attribution window, and filters, then reproduce both metrics side by side to pinpoint the exact divergence. You pick the correct definition based on the decision it supports, then you standardize it in a shared metric layer with tests, including join coverage checks and monitored drift in the share of filtered queries.

Sample Answer

Most candidates default to comparing pre and post retention curves, but that fails here because the unit of analysis changed, so the metric is not comparable. You run a bridge analysis where you compute retention under both identity systems during an overlap window, then estimate the mapping rate, collision rate, and split rate, like one user_id mapping to multiple device_ids and vice versa. You quantify how much retention changes purely from identity by holding behavior fixed, for example recomputing retention on the same events with both keys and reporting $\Delta$ attributable to keying. If the entire drop is explained by mapping artifacts, you fix stitching and backfill, otherwise you proceed to product and seasonality hypotheses.