ML System Design questions dominate senior engineer interviews at Meta, Google, Amazon, and Netflix. Unlike coding challenges that test algorithmic thinking, these questions evaluate your ability to architect production ML systems that serve millions of users. Expect 45-60 minute sessions where you design everything from recommendation engines to fraud detection pipelines.
What makes these interviews brutal is the sheer scope of decisions you must navigate under time pressure. Take designing YouTube's recommendation system: you need to choose between collaborative filtering and deep learning approaches, decide on batch versus real-time feature computation, architect for 2 billion users with sub-200ms latency, plan A/B testing strategies, and design monitoring for concept drift. One weak link in your reasoning can derail the entire discussion.
Here are the top 30 ML system design questions organized by the core competencies that separate senior engineers from the rest.
Problem Formulation & Requirements Gathering
Interviewers start here because they want to see if you can translate vague business requirements into concrete ML problems. Most candidates jump straight into model architectures without understanding what they're actually optimizing for, which signals junior-level thinking.
The trap is assuming every business problem needs machine learning. A Netflix interviewer once told me that the best answer they heard for 'design a system to reduce content delivery costs' was 'this isn't an ML problem, it's a CDN optimization problem.' Know when to say no.
Data Pipeline Design & Feature Engineering
Feature engineering separates production ML systems from academic projects, yet candidates consistently underestimate its complexity. You're not just building features, you're designing data pipelines that must handle billions of events, compute aggregations in real-time, and maintain consistency between training and serving.
The killer detail interviewers look for is understanding training-serving skew. If your Uber surge pricing model trains on batch-computed 'rides in last hour' features but serves with real-time counts, your model will fail in production. Always think through the end-to-end data flow.
Model Selection, Training & Offline Evaluation
This is where candidates reveal whether they've actually trained production models or just followed online tutorials. Interviewers probe your understanding of dataset construction, evaluation methodology, and the tradeoffs between model complexity and serving requirements.
A common failure mode is proposing complex architectures without justifying them. When a Google interviewer asks about model choice for query understanding, saying 'transformer because it's state-of-the-art' shows you don't understand the 10ms latency budget that rules out most deep learning approaches.
System Architecture & Model Serving
System architecture questions test whether you can bridge the gap between ML research and production engineering. The challenge is designing systems that serve models at scale while meeting latency, throughput, and reliability requirements that would make most data scientists uncomfortable.
Candidates often design systems that work in theory but crumble under real-world constraints. Proposing to serve a 500MB recommendation model for every user request shows you've never calculated memory requirements for 10,000 QPS. Always run the numbers.
Online Experimentation & A/B Testing
Online experimentation is where your ML system meets actual users, making it the ultimate test of production readiness. Interviewers focus here because A/B testing ML models involves unique challenges like network effects, long-term metrics, and statistical power that don't exist in traditional software testing.
The nuance that trips up most candidates is understanding when metrics diverge between offline evaluation and online experiments. If your offline AUC improves but online engagement drops, you need to diagnose whether it's a metric mismatch, data leakage, or fundamental model issues.
Monitoring, Debugging & Continuous Retraining
Production ML systems degrade silently, making monitoring and maintenance the difference between reliable products and spectacular failures. Interviewers dig into this because it reveals whether you understand that deploying a model is just the beginning of the ML lifecycle.
The insight that impresses senior engineers is recognizing that model performance degradation often has nothing to do with the model itself. When Uber's ETA predictions suddenly become less accurate, the cause might be a new road closure data source, a feature pipeline bug, or seasonal traffic pattern changes.
How to Prepare for ML System Design Interviews
Draw the data flow first
Before discussing any models, sketch how data flows from user actions to features to predictions to user-visible changes. Interviewers immediately spot candidates who haven't thought through the end-to-end pipeline.
Always estimate scale and latency
Calculate requests per second, feature store lookup times, and model inference latency with actual numbers. Saying 'we need sub-100ms latency' without breaking down where those milliseconds go shows surface-level thinking.
Propose specific metrics and thresholds
Instead of saying 'we'll monitor model performance,' specify 'we'll trigger retraining when 7-day rolling NDCG@10 drops below 0.85 or when feature drift exceeds 2 standard deviations.' Concrete numbers demonstrate production experience.
Practice explaining tradeoffs out loud
Record yourself explaining why you'd choose gradient boosting over neural networks for a specific use case. The ability to articulate technical tradeoffs clearly separates strong candidates from those who just memorize architectures.
Study real system architectures
Read engineering blogs from Netflix, Uber, and Meta about their recommendation systems, ranking models, and ML platforms. Reference specific techniques like 'Netflix's two-stage retrieval-ranking architecture' to show genuine industry knowledge.
