System Design Interview Questions

Sample Answer

You should ask these five questions in roughly this order: (1) What is the acceptable latency for a prediction, because sub-100ms changes your serving infrastructure entirely? (2) What is the expected QPS at peak, since Uber ride requests spike during rush hours and events? (3) Are predictions per-user, per-route, or per-region, because this determines your feature store design and cache strategy? (4) How frequently does the model retrain, since this affects whether you need online learning or batch retraining pipelines? (5) What is the fallback behavior if the model is unavailable, because this defines your reliability requirements? Each question narrows the design space so you avoid building something that is technically impressive but misaligned with actual constraints.

Sample Answer

You could structure requirements as a flat list or separate them into functional versus non-functional with explicit priority tiers. The tiered separation wins here because it forces you to distinguish what the system does (semantic search over documents, answer generation with citations, access control per user role) from how well it must do it (p99 latency under 2 seconds, retrieval recall above 0.9, cost per query under $0.01). For capacity estimation on 10M documents, calculate your vector index size: if each document averages 5 chunks at 1536 dimensions in float32, that is $10M \times 5 \times 1536 \times 4\ \text{bytes} \approx 307\ \text{GB}$, which tells you whether the index fits in memory or requires distributed search. Presenting requirements this way signals to the interviewer that you understand the tradeoffs that will drive every downstream architectural decision.

Sample Answer

You could version via URL path (e.g., /v1/recommendations, /v2/recommendations) or use a model version header (X-Model-Version) with a stable URL. Path versioning wins here because it makes routing, caching, and monitoring trivially separable per version, which matters when you are A/B testing models with different output schemas. Behind the versioned path, you maintain a contract registry (like protobuf or JSON Schema) that enforces additive-only changes within a major version: new fields are fine, removing or renaming fields forces a version bump. You deprecate old versions with a Sunset header and a minimum 30-day window, giving downstream consumers time to migrate.

Sample Answer

First, you need to think about what the caller knows at request time: entity type (driver, rider, trip), entity IDs, and which feature groups they need. So your request schema is POST /v1/features/fetch with a body like {entity_type, entity_ids: [], feature_groups: []}. Next, the response needs to handle partial availability since some features may be stale or missing, so you return a map of entity_id to feature vectors with a metadata block per feature indicating freshness timestamp and a status enum (FOUND, STALE, MISSING). For batches, you cap entity_ids at something like 500 and return a 400 if exceeded, rather than silently truncating. Finally, you version feature group schemas independently from the API version using a schema registry, so a model trained on feature_group v3 can explicitly request that version and not silently receive v4 with a changed feature definition.

Sample Answer

Start by thinking about the access pattern: writes are batch and periodic, but reads are point lookups by entity key (rider ID, driver ID) at very high QPS during inference. That means you want a key-value store optimized for read throughput, something like Redis for an online serving layer backed by a durable store like DynamoDB or Apache Cassandra for persistence. You would model features as a wide row keyed by entity ID with columns for each feature, and use a dual-write pattern where the batch pipeline writes to the durable store and then populates the cache. For freshness, you version each feature row with a batch timestamp so the serving layer can validate it has not gone stale, and you set TTLs in Redis that align with your hourly batch cadence plus a buffer.

Sample Answer

This question is checking whether you can distinguish between analytical workloads and serving workloads, then pick the right storage accordingly. Interaction logs for model training are consumed in large sequential scans, not random point lookups, so columnar Parquet files on S3 (queried via Spark, Athena, or Presto) are the clear choice: they offer excellent compression on repetitive event schemas, cost roughly $0.023/GB/month versus DynamoDB's far higher storage and read costs at this scale, and they integrate natively with ML training pipelines. You would partition by date and event type so training jobs can efficiently read only the slices they need. DynamoDB would only make sense if you needed low-latency random access to individual events, which is not the pattern here.

Sample Answer

This question is checking whether you can reason about cache hit rates in the context of power-law distributions, which dominate recommendation workloads. You should recognize that a small fraction of items (popular content, trending creators) account for a disproportionate share of embedding lookups, so caching the top $k$ most-requested embeddings in an in-memory store like Redis or Memcached gives you high hit rates with modest memory. You would use an LRU or LFU eviction policy, and for the long tail of cold embeddings, you serve directly from the embedding store (e.g., a sharded key-value database). The key tradeoff to articulate: cache staleness versus freshness, since embeddings may update after retraining, so you need a TTL or an invalidation mechanism tied to your training pipeline's publish cycle.

Sample Answer

The standard move is round-robin or least-connections load balancing across a fleet of stateless feature serving nodes backed by a distributed cache. But here, tail latency matters because you have a hard p99 target of 10ms, so you should use a load balancing policy that accounts for server response time, such as least-outstanding-requests or a "join the shortest queue" approach, which avoids sending traffic to nodes experiencing GC pauses or slow disk reads. You also want to add a local in-process cache (like a Guava or Caffeine cache in Java) on each serving node for the hottest features, reducing the round trip to the distributed store entirely. Hedged requests, where you send the same request to two nodes and take the first response, are another tool to shave p99 when the cost of redundant reads is acceptable.

Sample Answer

The standard move is to treat the model registry as an immutable artifact store (like MLflow or Vertex AI Model Registry) where every model version is a versioned, content-addressed artifact with metadata including training data snapshot, hyperparameters, and evaluation metrics. But here, multi-team governance matters because without strict promotion stages (dev, staging, canary, production) and ownership metadata, one team's bad deploy can poison another team's dependent pipeline. You should enforce that every model artifact must pass automated validation gates (performance thresholds on a holdout set, latency benchmarks, bias checks) before it can be promoted. A/B testing integration means the registry must expose version identifiers that your traffic routing layer (e.g., Envoy with header-based routing) can reference, so experiment configs map directly to registry versions. Rollback is then just re-promoting the previous version's artifact, which is already immutable and cached in your serving layer.

Sample Answer

Get this wrong in production and you silently approve thousands of fraudulent transactions before anyone notices. The right call is to build a monitoring pipeline with three layers: input drift detection (comparing incoming feature distributions against training baselines using statistical tests like PSI or KS tests), prediction drift detection (tracking shifts in output score distributions), and outcome drift detection (comparing predicted labels against delayed ground truth labels as they arrive). You should compute these metrics on rolling windows (e.g., hourly and daily) and feed them into an alerting system with tiered responses: a warning triggers a Slack alert and dashboard flag, a critical threshold triggers automatic fallback to a simpler, more stable model or a rule-based system. The retraining pipeline should be semi-automated, where drift detection triggers data collection and retraining jobs, but promotion to production still requires a human approval gate given the high stakes of fraud decisions.

Sample Answer

Get this wrong in production and a single GPU failure wastes hours of compute across hundreds of machines, which at Meta's scale translates to tens of thousands of dollars per incident. The right call is to combine elastic training with asynchronous checkpointing and hot spares. You configure the training framework (like TorchElastic) to detect node failures, remove the dead worker from the communication group, and continue training with $N-1$ workers while a hot spare joins and syncs. You checkpoint to persistent storage asynchronously every $k$ steps so the worst-case rollback is $k$ steps, not an entire epoch. The key detail is making your all-reduce topology reconfigurable at runtime so the ring or tree can be rebuilt without a full restart.

Sample Answer

Retry with naive logic sounds reasonable but breaks under partial failures because the generation service might succeed, charge the user, then the response gets lost on the way back, causing a retry that charges again. A saga pattern with compensating transactions doesn't work well here because you cannot "undo" a generated response already streamed to the user. That leaves idempotency keys as your primary mechanism: you assign a unique request ID at the API gateway, propagate it through all three services, and each service uses it to deduplicate. The charging service stores the idempotency key in a transactional write alongside the charge record, so any retry with the same key is a no-op. You combine this with a dead-letter queue for requests that fail mid-chain, allowing async reconciliation without blocking the user.