ML System Design Interview Questions

Sample Answer

The optimization target should be a proxy metric that is both measurable in the short term and causally linked to long-term retention, such as the number of quality streaming hours per week or the diversity of content engaged with over a 28-day window. Directly optimizing for churn (a binary label of whether a user cancels) is tempting but problematic: churn labels are sparse, delayed by billing cycles, and conflate price sensitivity with content dissatisfaction. You should propose a layered metric strategy where your offline objective (e.g., predicting $P(\text{active\_days} \geq k \mid \text{user, time\_window})$) is validated against the true business KPI of monthly churn rate through periodic A/B tests.

Sample Answer

You could treat this as an ML problem (build a model to predict which notifications a user will engage with and only send high-probability ones) or a product/engineering problem (fix notification timing, frequency capping, or deep-link behavior). The product investigation wins as the first step here because declining engagement often stems from notification fatigue or broken UX, and adding a ranking model on top of a broken delivery pipeline just adds complexity without addressing root cause. You should propose a diagnostic phase: segment engagement drops by platform, notification frequency, and time-since-follow, then only introduce ML (e.g., a send-time optimization model or a relevance scorer) once you confirm the infrastructure is sound and the remaining variance is genuinely user-preference-driven.

Sample Answer

You could compute this in a batch pipeline that runs hourly or in a streaming pipeline that updates continuously. Streaming wins here because the feature depends on a one-hour window that shifts constantly, and a batch job with hourly granularity would produce stale values for most of the hour, degrading recommendation quality. With a streaming approach using Apache Flink, you maintain two sliding window counters per user (1-hour and 24-hour), emit the ratio $m = \frac{c_{1h}}{c_{24h} + \epsilon}$ on each update, and write it to a feature store for online serving. You still want a batch backfill job over historical logs to generate consistent training labels, ensuring your offline and online computations use identical logic.

Sample Answer

First, you should check whether the training pipeline and serving pipeline compute the feature using the same code path and the same data source, because duplicated logic is the most common cause of skew. Next, verify the time semantics: if training uses a point-in-time correct snapshot but serving computes the average using a slightly different window boundary or includes the current day's partial data, the distributions will diverge. Then compare summary statistics ($\mu$, $\sigma$, percentiles) of the feature logged at serving time against the same feature in your training dataset for the same time period. The fix is to unify computation by having a single feature transformation that writes to both the feature store (for serving) and the training data generator (via log-and-replay or a shared feature store with time-travel capability). Adding automated data validation checks, such as distribution drift alerts using KL divergence or PSI, prevents this from silently recurring.

Sample Answer

First, you need to respect the temporal ordering, so you split by time: train on weeks 1 through 6, validate on week 7, and test on week 8, never shuffling randomly since that would leak future information into training. Next, you should consider whether your model generalizes across regions by stratifying your evaluation to report metrics per city or zone, because a model that looks great in aggregate might fail badly in sparse markets. For metrics, you would track MAE and RMSE at minimum, but also $P_{50}$ and $P_{90}$ absolute errors since ETA predictions have asymmetric user impact: underestimates cause more frustration than overestimates. Finally, you should compute these metrics on peak hours versus off-peak separately, because temporal distribution shift is the main failure mode in production.

Sample Answer

This question is checking whether you can resist over-engineering when labeled data is scarce and instead build a strong baseline first. You should start with a logistic regression or LightGBM model using handcrafted features: user's historical completion rate, genre affinity scores, time since last session, and content metadata like episode count and genre. With only 3 weeks of labels for new content, fine-tuning a large pretrained model risks severe overfitting, and you won't have enough signal to validate that it generalizes. Use the simple baseline's performance on a time-held-out split as your bar, then incrementally test whether adding pretrained content embeddings as input features to the simple model improves offline metrics like $\text{log-loss}$ and $\text{PR-AUC}$ before investing in end-to-end fine-tuning.

Sample Answer

This question is checking whether you can reason about the throughput vs. latency tradeoff in GPU-based serving. Server-side dynamic batching (as in systems like Triton Inference Server) groups incoming requests into a single batch to maximize GPU utilization, which increases throughput but adds queuing delay. You would batch when throughput is the bottleneck and your latency SLA has enough headroom to absorb the wait time, typically setting a max batch size and a max wait window (e.g., 5ms). For latency-critical paths where $p99$ latency must stay under, say, 20ms, you may prefer individual inference or very small batches. The right answer depends on your traffic pattern: bursty traffic benefits more from batching, while steady low-QPS traffic does not.

Sample Answer

The standard move is to use canary deployments where you route a small percentage (e.g., 1 to 5%) of traffic to the new model version while monitoring key metrics like engagement rate and latency. But here, the exception matters: if your model affects a revenue-critical surface, you also need shadow mode testing first, where the new model runs in parallel and its predictions are logged but not served, so you can compare outputs offline before any live exposure. You should version every model artifact with metadata (training data snapshot, hyperparameters, evaluation metrics) in a model registry, and your serving layer should support instant rollback by routing 100% of traffic back to the previous version via a configuration change, not a redeployment.

Sample Answer

The standard move is user-level randomization, where you assign each user to control or treatment independently. But here, network interference matters because a rider in treatment gets matched with a driver who may also be in the experiment, creating spillover effects that bias your estimates. You should randomize at the geo-time level: split geographic regions or hexagonal cells into treatment and control during specific time windows, ensuring that all participants within a cell-time block see the same algorithm. This cluster-based design sacrifices some statistical power, so you need to account for intra-cluster correlation when computing your variance estimates, effectively inflating your required sample size by the design effect factor $1 + (m-1)\rho$, where $m$ is the average cluster size and $\rho$ is the intra-cluster correlation.

Sample Answer

Get this wrong in production and you optimize for clickbait thumbnails that erode user trust and long-term engagement. The right call is to flag that CTR alone is a vanity metric here: if users click more but do not watch more, the thumbnails may be misleading, which will eventually hurt satisfaction and retention. You should recommend extending the test and adding guardrail metrics like streaming hours per session, bounce rate after click, and member satisfaction survey scores. Only ship if the CTR gain holds without degrading these downstream quality signals, or if you can demonstrate through a causal chain that the flat streaming hours are explained by a ceiling effect rather than disappointment.

Sample Answer

Get this wrong in production and you either waste compute retraining when nothing has changed, or you let a stale model serve bad ETAs for days during a distribution shift like a holiday or weather event. The right call is a hybrid approach: use a time-based schedule as a baseline (e.g., daily or weekly) combined with performance-based triggers that kick off emergency retraining when your online MAPE on a rolling window exceeds a threshold like $1.15 \times \text{baseline MAPE}$. Pure data-drift triggers using something like PSI or KS tests on feature distributions are useful as early warnings but should not be the sole trigger, because not all drift degrades performance. You also need guardrails: every retrained model must pass validation on a holdout set and a shadow serving period before promotion, so a bad training run on corrupted data does not automatically go live.

Sample Answer

Blaming a logging bug sounds reasonable but breaks under the fact that your offline metrics genuinely improved on held-out data. Attributing it to random variance does not work because a sustained CTR drop post-deployment is a clear signal. That leaves a train-serving skew or an offline-online metric mismatch as the most likely culprits. You should first check for feature skew by comparing the feature value distributions your model sees at serving time against what it saw during training, since even small discrepancies in image embedding pipelines or feature freshness can tank online performance. If features check out, the issue is likely that your offline evaluation did not capture a critical online dynamic like position bias, exploration effects, or the fact that users who click a misleading thumbnail churn, meaning your offline proxy metric was not aligned with the true business objective.