MLOps & Deployment Interview Questions

Sample Answer

Your first move is to profile whether the latency spike is in preprocessing, model inference, or network/serialization, because optimizing the wrong stage wastes effort. Once you isolate the bottleneck (usually inference for a 500MB transformer), you should apply model optimization techniques: convert to ONNX Runtime or TensorRT for faster GPU execution, enable dynamic batching to amortize overhead across concurrent requests, and consider quantizing the model from FP32 to INT8 which can cut inference time by 2-4x with minimal accuracy loss. On the infrastructure side, check if garbage collection pauses or cold container startups are causing the tail latency, and ensure your autoscaler is proactive rather than reactive. Tail latency often comes from a small fraction of requests hitting unwarmed instances or contending for shared resources.

Sample Answer

You could use a dedicated model server like TensorFlow Serving or embed the model directly in your application process. A dedicated model server wins here because it decouples model lifecycle management from application code, letting you update models independently, run A/B tests via traffic splitting, and leverage built-in batching and hardware acceleration without modifying your service. The 50ms budget is achievable with TF Serving or Triton over gRPC since the network hop within the same cluster adds only 1-2ms. Embedding the model only makes sense if you are serving a very small model (like a decision tree) where the serialization overhead of a network call would dominate inference time, which is not the case for most fraud detection models.

Sample Answer

Start by thinking about the compute requirements: a 3GB multi-modal model at 10K RPS means you cannot serve this on a single instance, so you need horizontal scaling behind a load balancer. Next, consider that preprocessing image and text inputs is often the bottleneck, so you should separate preprocessing into its own microservice or use async preprocessing pipelines to keep GPU utilization high on the inference nodes. For the model itself, you would deploy on GPU instances using Triton Inference Server with dynamic batching enabled, because batching transforms your throughput from hundreds to thousands of RPS per GPU by amortizing the fixed cost of kernel launches. Finally, you would use model parallelism or distillation if a single GPU cannot hold the 3GB model in memory alongside the batch, and implement request queuing with timeout policies so that under load you degrade gracefully rather than cascading failures.

Sample Answer

You could do scheduled retraining on a fixed cadence (e.g., daily or weekly) or event-driven retraining triggered by monitoring signals like data drift or performance degradation. Event-driven wins here because it avoids wasting compute when data is stable and reacts faster when distribution shifts actually occur. In practice, you should implement both: use a drift detector (e.g., monitoring feature distributions with KS tests or PSI) as the primary trigger, but keep a maximum staleness threshold as a fallback so the model never goes too long without retraining. This hybrid approach gives you cost efficiency and safety.

Sample Answer

Let me reason through this step by step. First, offline metrics alone are insufficient, so your pipeline needs a staged deployment gate: after offline validation passes, you deploy to a canary or shadow environment serving a small percentage of traffic. Second, you instrument online metrics (click-through rate, latency, error rates) and run a statistical test comparing the new model against the current champion, ideally an A/B test with a predefined significance threshold. Third, if the online metrics degrade beyond your tolerance (say, $p < 0.05$ on a one-sided test for regression), the pipeline should automatically roll back by re-routing 100% of traffic to the previous model version. Finally, you store every model artifact with its version and deployment metadata so rollback is a simple pointer swap, not a rebuild.

Sample Answer

First, you would compare the distribution of each feature at serving time against the distribution seen during training by logging serving-time feature values and computing divergence metrics like PSI (Population Stability Index) or KL divergence: $$D_{KL}(P \| Q) = \sum_{i} P(i) \log \frac{P(i)}{Q(i)}$$. Once you identify the skewed feature, you trace its lineage to check whether the transformation code differs between the training pipeline and the serving path, which is the most common root cause. Next, you check for subtler issues: whether the feature depends on data that arrives with different latency in batch vs. online (for example, a "last 7 days" window that uses complete data in training but partial data at serving time due to ingestion lag). Finally, you verify that any preprocessing steps like normalization or imputation use the same statistics (mean, variance) derived from training data rather than being recomputed on serving data, since recomputation introduces distribution drift that masquerades as skew.

Sample Answer

This question is checking whether you can manage feature evolution in production without causing downtime or data inconsistency. You run the backfill as an isolated batch job against your offline store (e.g., Spark over S3/GCS), writing the new feature column into a versioned feature group that is decoupled from the currently serving feature set. Once backfill completes and passes validation (null rate checks, distribution sanity, join key coverage), you retrain the model with the new feature included, then promote both the updated model and the new feature's online materialization pipeline together as an atomic deployment. This way your current serving pipeline is untouched until the new model version is fully validated, and you avoid the dangerous state where a model expects a feature that is not yet available online.

Sample Answer

This question is checking whether you can connect the dots between model serving and the full upstream provenance chain, not just whether you know what a model registry is. You would tag every deployed model version with a unique identifier that maps back to a lineage record containing the Git commit SHA, the dataset version or partition identifier, the experiment run ID from your tracking system, and the training pipeline execution ID. At inference time, you log the model version ID alongside each prediction so that any audit query can join predictions to their full lineage. You store this lineage metadata in a queryable store (a metadata database or a tool like MLflow's registry combined with a lineage service like Marquez or Amundsen) and enforce that no model can transition to a "production" stage in the registry without a complete lineage record passing validation.

Sample Answer

The standard move is to compare offline metrics like AUC or NDCG and pick the winner. But here, production readiness factors matter because two models with similar accuracy can behave very differently in deployment. You should track inference latency (p50, p95, p99), model size and memory footprint, prediction distribution stability compared to the previous version, fairness metrics across user segments, and resource cost per prediction. You would also compare training time and data freshness requirements, since a model that needs twice the compute to retrain on the same schedule may not be worth a marginal accuracy gain. Finally, check if either version shows signs of shortcut learning by examining feature importance drift relative to prior versions.

Sample Answer

The standard move is to use Population Stability Index (PSI) for categorical or binned continuous features because it is symmetric, easy to interpret, and widely adopted, with $PSI < 0.1$ indicating no significant drift, $0.1 \leq PSI < 0.25$ as moderate, and $PSI \geq 0.25$ triggering an alert. But here, the nature of your features matters because KS test works better for continuous features where you care about the maximum pointwise difference between CDFs, and KL divergence is useful when you need a probabilistic interpretation but is asymmetric and sensitive to zero bins. For a fraud model specifically, you should layer these statistical tests with business-level monitors like alert rate and approval rate, because feature drift alone does not always cause performance degradation. Set your alerting thresholds empirically by backtesting against known incidents rather than relying solely on textbook cutoffs.

Sample Answer

Get this wrong in production and you silently degrade experience for an entire user segment for weeks before anyone notices. The right call is to monitor sliced metrics, not just aggregates: you should compute precision, recall, and F1 per class or per important segment (language, region, device) and set per-slice alerting thresholds. Your observability stack should include a dashboard with automated slice-level performance tracking, a data quality layer that flags volume shifts per class (e.g., if one language suddenly gets 3x more traffic), and prediction distribution monitors that detect when the model's confidence calibration shifts for specific slices. You should also log model inputs and outputs to a queryable store so you can retroactively debug which examples the model got wrong, and set up conditional alerts like: if any single-class precision drops more than 5 percentage points over a rolling 48-hour window, page the on-call engineer.

Sample Answer

Get this wrong in production and you either blow your budget within weeks or crater your model's serving throughput, causing upstream services to timeout. The right call is to start by profiling your workload: measure actual GPU utilization per instance, batch sizes, and request latency distributions. You will likely find that $p4d$ instances are underutilized for most queries, so you should tier your traffic by routing simple requests to cheaper Inferentia2 chips ($ml.inf2.xlarge$) and reserving $p4d$ instances only for complex, long-context requests. Layer on Savings Plans or Reserved Instances for your baseline capacity, use spot instances for burst traffic, and apply model optimizations like quantization (INT8 or FP8) to reduce the GPU memory footprint, letting you serve more concurrent requests per node.

Sample Answer

A single monolithic container sounds reasonable but breaks under independent scaling needs: if your audio decoding step is CPU-bound and your embedding generation is GPU-bound, you cannot scale them separately, wasting resources on both. Running everything as fully decoupled microservices does not work because the inter-service communication overhead for large audio tensors introduces significant latency and serialization cost. That leaves a middle path: group tightly coupled steps (decode and normalize) into one container and isolate the GPU-heavy embedding computation into a second container, connected via a lightweight message queue or shared volume. This gives you independent scaling for the GPU-intensive stage while keeping data transfer minimal between logically related steps.