Design a Recommendation System

Problem Formulation

Before you write a single line of model code, you need to nail down what you're actually building. Recommendation systems look deceptively similar on the surface, but "recommend videos on Netflix" and "recommend products on Amazon" have fundamentally different latency requirements, feedback signals, and failure modes. The first question out of your mouth should be: what are we recommending, and to whom?

Start by pinning the product context. Is this a personalized homepage feed, a "more like this" carousel, an email digest, or a search re-ranking surface? Each one changes everything downstream. A homepage feed needs to be fresh and fast (the user is staring at a loading spinner). An email digest can tolerate a batch job that runs at 2am. A "more like this" panel is semi-real-time but anchored to a specific item, which shifts the retrieval strategy entirely. Get the interviewer to commit to one before you go further.

Clarifying the ML Objective

ML framing: Given a user and their context, rank a set of candidate items by predicted probability of a target engagement action (click, watch, purchase) so the top-K results maximize both user satisfaction and business value.

The business goal and the ML objective are almost never the same thing, and interviewers specifically want to see you acknowledge that gap. The business wants revenue or long-term retention. What you can actually train a model to predict is something measurable and immediate: did the user click? Did they watch more than 30 seconds? Did they add to cart? These proxy metrics are your ML targets, but they're imperfect stand-ins for the real goal.

CTR is the classic trap. A model optimized purely for clicks will learn to recommend clickbait. Watch time is better for video, but a model maximizing watch time might surface addictive content over genuinely satisfying content. This is why production systems almost always use multi-task learning, jointly optimizing a satisfaction signal (rating, save, share) alongside an engagement signal. Bring this up proactively. It signals you've thought beyond the toy problem.

The ML task itself is a ranking problem at its core. You're not predicting a single outcome in isolation; you're ordering a set of candidates for a given user. That means your loss functions, evaluation metrics, and even your data collection strategy need to reflect the relative nature of the problem, not just pointwise prediction accuracy.

Functional Requirements

Core Requirements

• The system must generate a personalized ranked list of K items (K=10 to 50 depending on surface) for a given user ID and request context, with p99 end-to-end latency under 200ms.
• The model must accept user behavioral history (implicit signals: clicks, views, skips), item metadata (content embeddings, category, popularity), and session context (what the user just interacted with) as inputs.
• The system must handle new users with no interaction history (cold-start) by falling back to content-based or popularity-based signals without a hard failure.
• Recommendations must reflect reasonably fresh signals: a user who just watched three horror films should see more horror within the same session, not the next day.
• The system must support offline A/B evaluation and online experimentation, allowing multiple model variants to serve different user buckets simultaneously.

Below the line (out of scope)

• Real-time personalization within a single page scroll (sub-50ms streaming updates as the user scrolls).
• Explicit content moderation or safety filtering (assume a separate pipeline handles this upstream).
• Cross-surface recommendations (e.g., using mobile behavior to personalize desktop feed) unless the interviewer specifically asks.

Metrics

Offline metrics

Recall@K measures whether the items a user actually engaged with appear in your top-K predictions. It's the right primary metric here because you care about coverage: did you surface the good stuff at all? NDCG@K (Normalized Discounted Cumulative Gain) goes further by rewarding you for ranking the relevant items higher, not just including them. Use NDCG@K when position matters, which it almost always does since users rarely scroll past the first few results.

AUC-ROC is useful during model development as a threshold-independent measure of how well your model separates positive from negative examples. But don't lead with AUC in the interview. It's a pointwise metric and doesn't capture the ranking quality that actually matters for recommendations.

For multi-task models, track per-task AUC separately. A model that improves CTR prediction while degrading satisfaction prediction is not a net win, and you won't see that if you only report a combined loss.

Online metrics

• Click-through rate (CTR): the primary engagement signal, easy to measure, but gameable.
• Watch time / session depth: stronger signal of genuine satisfaction for video or content surfaces.
• Conversion rate / revenue per session: the north star for e-commerce recommendations.
• Return visit rate: a longer-horizon metric that captures whether recommendations are building habit or burning trust.

Guardrail metrics

These are the metrics you set thresholds on and refuse to regress, even if your primary metric improves.

• p99 serving latency: if you blow the 200ms budget, the product team will roll you back regardless of NDCG.
• Coverage: the fraction of the item catalog that gets recommended to at least one user. A model that only recommends the top 1% of popular items is failing the business even if its CTR looks great.
• Intra-list diversity: average pairwise dissimilarity within a returned list. Users who get ten nearly identical recommendations churn.
• Demographic parity or exposure fairness: if the system is systematically under-recommending items from certain creators or categories, that's a legal and reputational risk.

Tip: Always distinguish offline evaluation metrics from online business metrics. Interviewers want to see you understand that a model with great AUC can still fail in production. A model trained on historical clicks will look great offline, then go live and optimize for the wrong behavior entirely. The gap between NDCG and actual watch time is where most recommendation projects die.

Constraints & Scale

Assume you're building for a mid-to-large consumer platform: 100 million monthly active users, an item catalog of 10 million items (videos, products, or songs), and peak traffic around 50,000 recommendation requests per second during primetime.

At 50K QPS with a 200ms end-to-end budget, you have roughly 20ms for feature fetching, 20ms for candidate retrieval, 100ms for ranking inference, and 60ms of buffer for network and orchestration overhead. That's tight. It's why the two-stage retrieval-ranking architecture exists: you cannot run a deep neural network over 10 million items in 100ms. You need retrieval to cut that down to a few hundred candidates first.

Cold-start is a hard constraint, not an edge case. On a platform with healthy growth, 10-20% of daily active users may have fewer than five interactions. Your design needs an explicit fallback strategy: content-based embeddings for new items, demographic or geographic priors for new users. If you don't mention this, the interviewer will ask, and "we'd handle that later" is not an acceptable answer at the senior level.

Feedback sparsity is the other structural constraint. Even active users only interact with a tiny fraction of the catalog. Your training data is overwhelmingly negative by default, which means your negative sampling strategy directly shapes what the model learns. This comes up in the model development section, but flag it here so the interviewer knows you're thinking about it.

Data Preparation

Most recommendation systems fail not because of bad models, but because of bad data. Before you write a single line of model code, you need to know where your training signal comes from, how clean it is, and whether your labels actually reflect what you're trying to predict.

Data Sources

You're pulling from four broad categories of data, and each one has a different reliability profile.

User activity logs are your richest signal. Every click, scroll, purchase, skip, and dwell-time event tells you something about preference. At Netflix scale, this is billions of events per day. At a mid-sized product, you might have tens of millions. The schema matters here: log the event type, user ID, item ID, timestamp, surface (homepage vs. search vs. email), and session ID. If you skip the surface, you'll lose the ability to train surface-specific models later.

Item catalog gives you the content side: titles, categories, tags, prices, publish dates, content embeddings. This data is usually smaller in volume (millions of items, not billions of events) but slower to update. A product catalog might refresh nightly. A news feed might need updates every few minutes.

Contextual signals are often underweighted by candidates. Time of day, device type, user location, and current session context (what the user just viewed) are powerful features. They're also ephemeral: you need to capture them at request time and log them alongside the interaction event, or they're gone.

Third-party data (demographic enrichment, social graph signals, external content metadata) adds coverage for cold-start but introduces reliability risk. External APIs go down. Data licensing changes. Treat third-party data as a supplement, not a foundation.

Interview tip: When an interviewer asks "what data would you use?", don't just say "user behavior." Walk through each source, what signal it provides, how fresh it needs to be, and what can go wrong. That specificity is what separates a senior answer from a junior one.

Event Logging Schema

Here's a concrete event schema worth sketching in your interview:

CREATE TABLE interaction_events (
    event_id        UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    user_id         UUID NOT NULL,
    item_id         UUID NOT NULL,
    event_type      VARCHAR(50) NOT NULL,     -- 'click', 'view', 'purchase', 'skip', 'like'
    dwell_time_ms   INTEGER,                  -- NULL for non-view events
    surface         VARCHAR(50) NOT NULL,     -- 'homepage', 'search', 'email', 'pdp'
    session_id      UUID NOT NULL,
    device_type     VARCHAR(20),              -- 'mobile', 'desktop', 'tv'
    position        INTEGER,                  -- rank position shown to user
    request_id      UUID,                     -- ties back to the recommendation request
    event_ts        TIMESTAMP NOT NULL,
    server_ts       TIMESTAMP NOT NULL        -- when the server received it (for late-arrival detection)
);

CREATE INDEX idx_events_user_ts ON interaction_events(user_id, event_ts DESC);
CREATE INDEX idx_events_session  ON interaction_events(session_id, event_ts);

Notice the position column. You need to log where in the ranked list the item appeared. Without it, you can't correct for position bias during training.

Late-arriving events are a real operational problem. Mobile clients buffer events and flush them when connectivity returns, sometimes hours later. Your ingestion pipeline needs to handle out-of-order events and use server_ts vs. event_ts to detect them. Flink's event-time processing with watermarks is the standard approach here.

---

Label Generation

This is where most candidates get vague, and where interviewers push hardest.

Explicit feedback (star ratings, thumbs up/down) is clean but sparse. Users rate maybe 1-2% of items they interact with. You can't train a model on that alone.

Implicit feedback is abundant but noisy. A click might mean genuine interest or accidental tap. A 30-second view on a 2-minute video is ambiguous. A completed view is a strong positive signal. You need to define thresholds that make sense for your product:

def assign_label(event: dict) -> int | None:
    """
    Returns 1 (positive), 0 (negative), or None (skip/ambiguous).
    Thresholds should be tuned per surface and content type.
    """
    if event["event_type"] == "purchase":
        return 1
    if event["event_type"] == "like":
        return 1
    if event["event_type"] == "view":
        completion = event.get("dwell_time_ms", 0) / event["item_duration_ms"]
        if completion >= 0.8:
            return 1   # strong positive
        if completion < 0.1 and event.get("dwell_time_ms", 0) < 3000:
            return 0   # likely accidental
        return None    # ambiguous middle ground
    if event["event_type"] == "skip":
        return 0
    return None

Delayed feedback is a subtle trap. For e-commerce, a user might click an item today and purchase it three days later. If your training pipeline runs daily, you'll miss that conversion and incorrectly label the click as a non-purchase. The fix is to use a label delay window: hold events in a staging area for 24-72 hours before finalizing labels.

Warning: Label leakage is one of the most common ML system design mistakes. Always clarify the temporal boundary between features and labels. If your label is "did the user purchase within 7 days of the click," then every feature you use must come from before the click timestamp, not from the 7-day window. Using post-click features as inputs is a leakage pattern that produces inflated offline metrics and models that fall apart in production.

Exposure bias is the hardest problem to explain cleanly in an interview, and doing so will impress your interviewer. Items that were never shown to a user can't be clicked. So your training data is not a random sample of user-item pairs; it's a biased sample of pairs the previous model decided to show. If you train naively on this data, you're training a model to replicate the existing model's biases, not to find genuinely good recommendations.

The standard mitigation is in-batch negative sampling: treat other items in the same training batch as negatives for a given user. This approximates the full item distribution rather than just the exposed items. For harder negatives, you can sample from items the user was shown but didn't interact with.

def build_training_pairs(
    positive_events: list[dict],
    item_pool: list[str],
    num_negatives_per_positive: int = 4,
    hard_negative_pool: list[dict] | None = None,
) -> list[dict]:
    """
    Constructs (user, item, label) training triples.
    Mixes random negatives with hard negatives (shown but not clicked).
    """
    pairs = []
    for event in positive_events:
        pairs.append({
            "user_id": event["user_id"],
            "item_id": event["item_id"],
            "label": 1,
            "source": "positive",
        })
        # Hard negatives: shown but not engaged
        if hard_negative_pool:
            hard_negs = [
                e for e in hard_negative_pool
                if e["user_id"] == event["user_id"]
            ][:2]
            for neg in hard_negs:
                pairs.append({
                    "user_id": event["user_id"],
                    "item_id": neg["item_id"],
                    "label": 0,
                    "source": "hard_negative",
                })
        # Random negatives from item pool
        random_negs = random.sample(item_pool, num_negatives_per_positive - len(hard_negs))
        for item_id in random_negs:
            pairs.append({
                "user_id": event["user_id"],
                "item_id": item_id,
                "label": 0,
                "source": "random_negative",
            })
    return pairs

---

Data Processing & Splits

Raw event logs are not training data. You need several cleaning passes before they're usable.

Deduplication and bot filtering come first. Deduplicate events by (user_id, item_id, event_type, session_id) within a short time window (say, 5 minutes) to collapse double-clicks. Filter bot traffic using request rate thresholds, user-agent patterns, and behavioral signals (inhuman scroll speed, no mouse movement variance). Bot events in your training data will teach your model to optimize for bot behavior.

Outlier removal matters for dwell-time features. A user who leaves a tab open for 8 hours isn't engaged; they forgot to close it. Cap dwell time at a reasonable maximum (3-5x the item's median consumption time). Similarly, users with thousands of interactions in a single day are almost certainly bots or test accounts.

For class imbalance: in most recommendation datasets, positive examples are 1-5% of all logged impressions. You have two options. Downsample negatives to a manageable ratio (1:4 or 1:10 positive:negative) during training, or use a weighted loss function. Downsampling is simpler and usually works well. If you downsample, remember to correct your model's predicted probabilities at serving time using the sampling rate.

Train/validation/test splits are where a lot of candidates make a critical mistake. Don't split randomly. Split by time.

If you split randomly, a user's Monday behavior ends up in training and their Sunday behavior ends up in test. The model has effectively seen the future. Your offline metrics will look great and your online metrics will disappoint.

The right approach: train on events before time T, validate on events between T and T+delta, test on events after T+delta. Choose T based on how much data you need and how far ahead you want to evaluate. A common setup for a daily-retrained model is 90 days of training data, 7 days of validation, 7 days of test.

def time_based_split(
    events: pd.DataFrame,
    train_end: str,
    val_end: str,
) -> tuple[pd.DataFrame, pd.DataFrame, pd.DataFrame]:
    """
    Splits events into train/val/test by timestamp.
    All timestamps are event_ts, not server_ts.
    """
    train_end_ts = pd.Timestamp(train_end)
    val_end_ts   = pd.Timestamp(val_end)

    train = events[events["event_ts"] < train_end_ts]
    val   = events[(events["event_ts"] >= train_end_ts) & (events["event_ts"] < val_end_ts)]
    test  = events[events["event_ts"] >= val_end_ts]

    return train, val, test

Data versioning is non-negotiable in production. Every training run should reference a specific, immutable snapshot of the training dataset. Store datasets in S3 or GCS with versioned prefixes, and log the dataset version alongside every model artifact in your model registry. When a model degrades in production, you need to be able to reproduce the exact training data that produced it.

A practical pattern: write datasets as date-partitioned Parquet files and register each snapshot in a data catalog (AWS Glue, Hive Metastore, or even a simple metadata table). Your training job takes a dataset_version argument, not a date range it computes itself.

---

---

Data Freshness Tradeoffs

The right pipeline architecture depends on how fast your data distribution moves.

For most recommendation systems, a batch pipeline (Spark jobs running hourly or daily) is sufficient for user behavior aggregates and item metadata. These features don't change fast enough to justify streaming infrastructure, and batch pipelines are far easier to debug and backfill.

Streaming pipelines (Kafka + Flink) are worth the operational cost for a narrow set of signals: what the user just viewed in the current session, real-time trending items, and inventory changes (an item going out of stock). These signals have a half-life measured in minutes, not hours.

Key insight: The most common mistake is building a streaming pipeline for everything because it feels more "real-time." In practice, a user's long-term taste profile doesn't change meaningfully in the last 10 minutes. Spend your streaming budget on signals that actually decay that fast: session context, trending content, and availability.

A hybrid architecture works well: batch pipelines populate the feature store with stable user and item features, while a lightweight streaming job writes session-level features to a separate Redis key with a short TTL. The ranking model reads from both at serving time.

Feature Engineering

Good features matter more than model architecture in most production recommendation systems. A simple logistic regression with well-engineered features will beat a transformer with raw IDs almost every time. Here's how to think about the feature space.

Feature Categories

User Features

These capture who the user is and what they've done historically. They're mostly precomputed in batch.

The user_embedding deserves special attention. For users with rich history, a learned ID embedding captures behavioral patterns that no hand-crafted feature can match. For new users with sparse history, fall back to demographic or content-based signals. You need both.

Item Features

Content embeddings (BERT, ResNet) are your cold-start lifeline. A brand-new item has zero interaction history, but you can still compute its content embedding immediately at publish time and use it for retrieval.

Contextual Features

These change with every request and can't be precomputed. They have to be assembled at serving time.

• hour_of_day (int, 0-23): user behavior shifts dramatically between 7am commute and 11pm wind-down
• day_of_week (int, 0-6): weekend vs. weekday consumption patterns differ
• device_type (categorical): mobile users want short-form; desktop users tolerate longer content
• session_item_ids (int[], last 10 items): what the user has already seen in this session, used to avoid repeats and capture intent
• request_surface (categorical): homepage feed vs. "more like this" vs. search results; the same user has different intent on each surface

Session context is underused by most candidates. The last 3-5 items a user interacted with in the current session are often the strongest signal for their immediate intent.

Cross Features and Interactions

Cross features combine user and item signals to capture affinity directly.

The key decision is where to compute these. user_category_affinity can be precomputed in batch since category preferences are stable over hours. recency_weighted_affinity that incorporates the last 10 minutes of session behavior needs to be computed at serving time or via a streaming pipeline.

Common mistake: Candidates often list features without explaining how they're computed or where they live. The interviewer wants to know: is this a batch feature in the feature store, a streaming feature, or computed at request time? Always say which.

---

Feature Computation

Batch Features

Most user and item features are stable enough to compute on a schedule. The pipeline looks like this:

Raw event logs (S3/GCS)
  → Spark/SQL job (hourly or daily)
  → Computed feature values
  → Written to offline store (Parquet) + online store (Redis)

A concrete example: computing 30d_genre_affinity for all users.

# PySpark batch job — runs hourly
from pyspark.sql import functions as F

genre_affinity = (
    events
    .filter(F.col("event_type").isin(["click", "watch", "save"]))
    .filter(F.col("event_ts") >= F.date_sub(F.current_date(), 30))
    .join(items.select("item_id", "genre"), on="item_id")
    .groupBy("user_id", "genre")
    .agg(
        F.sum(
            F.when(F.col("event_type") == "watch", F.col("watch_fraction") * 2.0)
             .otherwise(1.0)
        ).alias("weighted_score")
    )
    .withColumn(
        "normalized_score",
        F.col("weighted_score") / F.sum("weighted_score").over(
            Window.partitionBy("user_id")
        )
    )
)

# Write to feature store
genre_affinity.write.format("delta").mode("overwrite").save("s3://features/user_genre_affinity/")

Watch time gets a 2x weight over a bare click. That's a deliberate label design choice, not an accident.

Near-Real-Time Features

Some signals go stale in minutes. A user who just watched three horror movies in a row should get horror recommendations now, not after the next hourly batch run. This is where streaming comes in.

User interaction event
  → Kafka topic (user-events)
  → Flink job (sessionizes, aggregates over 10-minute window)
  → Redis (online feature store, TTL = 1 hour)

# Flink streaming job (simplified PyFlink)
env = StreamExecutionEnvironment.get_execution_environment()

events = env.add_source(KafkaSource("user-events"))

session_features = (
    events
    .key_by(lambda e: e["user_id"])
    .window(SlidingEventTimeWindows.of(
        Time.minutes(10), Time.minutes(1)
    ))
    .aggregate(SessionAggregator())  # computes item_ids, category counts
)

session_features.add_sink(RedisSink(
    key_pattern="session_features:{user_id}",
    ttl_seconds=3600
))

The TTL is important. Session features that are more than an hour old are probably misleading rather than helpful.

Real-Time Features

A small set of features can only be computed at the moment of the request. The most common example is the user's current query embedding (for search-adjacent surfaces) or the embedding of the item they're currently viewing (for "more like this").

# At serving time, inside the recommendation service
def build_request_features(user_id: str, context: RequestContext) -> Features:
    # Fetched from Redis — precomputed
    user_features = feature_store.get_user_features(user_id)
    session_features = feature_store.get_session_features(user_id)

    # Computed right now
    current_item_embedding = embedding_service.encode(
        context.current_item_id
    )  # cached if item was seen before

    return Features(
        user=user_features,
        session=session_features,
        context=ContextFeatures(
            hour_of_day=context.timestamp.hour,
            device_type=context.device,
            current_item_embedding=current_item_embedding,
        )
    )

Keep real-time computation minimal. Every millisecond you spend here comes out of your latency budget. If a feature can be precomputed, precompute it.

---

Feature Store Architecture

The feature store solves one problem: making sure the features your model sees at training time are identical to the features it sees at serving time.

The online store (Redis or DynamoDB) serves features at inference time. Single-digit millisecond lookups, key-value access by user ID or item ID. This is where your batch and streaming jobs write their outputs.

The offline store (Parquet files on S3, or a Hive/Delta table) serves training. It needs to support point-in-time correct lookups, meaning when you're building a training example for an event that happened on Tuesday at 3pm, you need the feature values as they existed at that moment, not today's values. Getting this wrong causes label leakage.

The shared transformation layer is what actually prevents skew. Both the batch training pipeline and the online serving code call the same Python functions to compute derived features. If you compute recency_weighted_affinity with a 7-day half-life in training but a 30-day half-life at serving time, your model will silently underperform and you'll spend weeks debugging it.

Key insight: The most common failure mode in ML systems is training-serving skew. Features computed differently in batch training vs. online serving cause your model to operate on a distribution it was never trained on. Always design for consistency: shared transformation code, shared feature definitions, and integration tests that compare offline and online feature values for the same entity at the same timestamp.

Tools like Feast and Tecton enforce this by design. You define a feature transformation once, and the framework handles materializing it to both the online and offline stores. If you're building from scratch, at minimum write unit tests that run your transformation logic against known inputs and assert the same output in both paths.

One more thing on encoding strategy. User and item ID embeddings learned end-to-end capture collaborative filtering signal beautifully for users and items with enough history. But they're useless for new users and new items because the IDs have never been seen during training. Pre-trained content embeddings (BERT for text, ResNet for images) cover this gap. In practice, you use both: the ID embedding when available, falling back to the content embedding otherwise, and sometimes concatenating both when the item has both history and rich content.

Model Selection & Training

Start simple. Before you touch neural networks, tell the interviewer you'd validate the problem with a popularity-based baseline: recommend the top-K trending items globally, optionally filtered by category. It sounds naive, but it's surprisingly hard to beat on cold-start users, and it gives you a concrete floor to measure against. If your fancy model can't outperform "just show what's popular," something is wrong with your training data or evaluation setup.

From there, the production architecture has two distinct stages, and you should treat them as separate problems with different constraints.

Model Architecture

The core insight is that you can't run a rich ranking model over your entire catalog. At 10 million items, even a 1ms model call per item takes 2.7 hours per request. So you split the problem: retrieval gets you from millions to hundreds fast, then ranking scores those hundreds carefully.

Stage 1: Candidate Retrieval

The retrieval stage needs to be approximate, cheap, and scalable. Your options, roughly in order of sophistication:

Rule-based filters are your zero-shot baseline. Trending items, geolocation-filtered content, recently published items. No training required, ships in a day, and handles new users gracefully.

Matrix factorization (ALS) is the classic step up. You factorize the user-item interaction matrix into latent vectors, then use ANN search at serving time. It works well when you have dense interaction data, but it can't incorporate side features natively, which hurts cold-start.

The two-tower network is what you'd propose for production. Two separate neural networks, one encoding the user and one encoding the item, trained jointly so their dot product approximates interaction probability.

The user tower takes: user ID embedding (dim 64-128), behavior sequence aggregates (mean-pooled embeddings of last N interacted items), and context features (time of day, device type). The item tower takes: item ID embedding, content embeddings from a pre-trained encoder (BERT for text, ResNet for images), and metadata features like category and age.

Both towers output a dense embedding of the same dimension, typically 64 or 128. During training, you compute the dot product between user and item embeddings and apply softmax over a batch of negatives. At serving time, you freeze the item tower, pre-compute all item embeddings, and load them into a FAISS index. Retrieval becomes a single ANN query.

class TwoTowerModel(nn.Module):
    def __init__(self, user_feature_dim: int, item_feature_dim: int, embedding_dim: int = 64):
        super().__init__()
        self.user_tower = nn.Sequential(
            nn.Linear(user_feature_dim, 256),
            nn.ReLU(),
            nn.Linear(256, embedding_dim),
            nn.LayerNorm(embedding_dim)
        )
        self.item_tower = nn.Sequential(
            nn.Linear(item_feature_dim, 256),
            nn.ReLU(),
            nn.Linear(256, embedding_dim),
            nn.LayerNorm(embedding_dim)
        )

    def forward(self, user_features, item_features):
        user_emb = self.user_tower(user_features)   # (batch, embedding_dim)
        item_emb = self.item_tower(item_features)   # (batch, embedding_dim)
        logits = torch.sum(user_emb * item_emb, dim=-1)  # dot product
        return logits, user_emb, item_emb

Loss function: in-batch softmax cross-entropy with sampled negatives. For each positive (user, item) pair in the batch, treat all other items as negatives. This is efficient and scales well. If you have explicit negative feedback (skips, dislikes), add those as hard negatives with higher weight.

The output is a ranked list of item IDs from the FAISS index, typically top 200-500 candidates passed downstream.

Common mistake: Candidates propose the two-tower model but forget to explain how the index gets built and refreshed. The interviewer will ask. Pre-build the FAISS index offline over all item embeddings, refresh it hourly or daily depending on catalog churn. For catalogs with high item turnover (news, live events), you need a streaming index update path.

Stage 2: Ranking

The ranking stage can afford to be slower and richer. You're scoring 200-500 candidates, not millions, so you can use cross-features that would be impossible to compute at retrieval scale.

Start with LightGBM as your baseline. It's fast to train, interpretable, handles missing features gracefully, and often matches deep models on tabular data. If you're in an interview and asked to justify your model choice, "LightGBM is a strong baseline that ships fast and is easy to debug" is a completely valid answer.

For production, propose a Wide & Deep network (or DCN v2 if you want to show depth). The wide component is a linear model over sparse cross-product features, capturing memorization: "users who liked X also liked Y." The deep component is an MLP over dense embeddings, capturing generalization to unseen combinations.

The ranking model input for each candidate: - User features: demographic embeddings, historical engagement rates by category, session context (last 5 items viewed) - Item features: content embedding (from retrieval tower), popularity stats, freshness score - Cross features: user-item affinity score (how often the user engages with this item's category), user's historical CTR on similar items

# Input contract for ranking model
# user_features:  Tensor of shape (batch_size, user_feature_dim=128)
# item_features:  Tensor of shape (batch_size, item_feature_dim=64)
# cross_features: Tensor of shape (batch_size, cross_feature_dim=32)
# Output: probability score in [0, 1] per candidate

class WideAndDeepRanker(nn.Module):
    def __init__(self, wide_dim: int, deep_input_dim: int):
        super().__init__()
        # Wide: linear on sparse cross features
        self.wide = nn.Linear(wide_dim, 1)
        # Deep: MLP on dense features
        self.deep = nn.Sequential(
            nn.Linear(deep_input_dim, 256),
            nn.ReLU(),
            nn.Dropout(0.3),
            nn.Linear(256, 128),
            nn.ReLU(),
            nn.Linear(128, 1)
        )

    def forward(self, wide_features, deep_features):
        wide_out = self.wide(wide_features)
        deep_out = self.deep(deep_features)
        return torch.sigmoid(wide_out + deep_out)

Loss function choice matters here. Binary cross-entropy works for CTR prediction, but if you're optimizing for watch time or purchase value, you want a regression head or a pairwise ranking loss like BPR (Bayesian Personalized Ranking), which directly optimizes the ordering of items rather than absolute scores.

If the interviewer asks about session-aware ranking (common at Spotify or Netflix), mention transformer-based sequential models. You encode the user's current session as a sequence of item embeddings, run a transformer over it, and use the output as the user representation. This captures "the user just watched three action movies, so rank action higher right now" in a way that static user embeddings can't.

Multi-task learning is worth raising proactively. If you optimize purely for CTR, you'll surface clickbait. If you optimize purely for watch time, you'll surface long videos regardless of quality. The practical answer is a multi-task model with separate output heads for each objective, sharing the lower layers, and combining the scores with learned or manually tuned weights at serving time.

# Multi-task output heads on shared representation
class MultiTaskRanker(nn.Module):
    def __init__(self, shared_dim: int):
        super().__init__()
        self.shared = nn.Sequential(
            nn.Linear(shared_dim, 256), nn.ReLU(),
            nn.Linear(256, 128), nn.ReLU()
        )
        self.ctr_head = nn.Linear(128, 1)         # click probability
        self.engagement_head = nn.Linear(128, 1)  # watch time / save rate

    def forward(self, x):
        h = self.shared(x)
        return torch.sigmoid(self.ctr_head(h)), torch.sigmoid(self.engagement_head(h))

Training Pipeline

For the retrieval two-tower, you need distributed training. A catalog of 10M items with in-batch negatives means large batch sizes (4096+) to get enough negatives per step. Use PyTorch DDP across 4-8 GPUs, or Ray Train if you're on a heterogeneous cluster. Training time is typically a few hours per run.

The ranking model is lighter. LightGBM trains on a single machine in minutes. The Wide & Deep model trains on a single GPU in under an hour for most datasets. Don't over-engineer the infrastructure here.

Retraining cadence depends on how fast your data distribution shifts. News recommendations go stale in hours. Movie recommendations might be stable for days. A reasonable default: retrain the ranking model daily on a rolling window of the last 30 days of interactions, and retrain the retrieval model weekly since embedding spaces are more stable. Incremental fine-tuning (continuing from the last checkpoint) is faster but risks catastrophic forgetting if the distribution shifts sharply.

Tip: When the interviewer asks about retraining, don't just say "retrain daily." Explain the tradeoff: full retraining is expensive but gives you a clean model; incremental fine-tuning is cheap but can accumulate bias from feedback loops. The right answer depends on your drift rate.

For hyperparameter tuning, propose Bayesian optimization (Optuna or Ray Tune) over grid search. The key hyperparameters to tune are embedding dimension, learning rate, batch size, and the loss weighting between multi-task heads. Run tuning on a 10% data sample first to find the viable range, then do a fine-grained search on full data.

Training data windowing is a judgment call. Using all historical data gives you more signal but older interactions may reflect a different user or a different catalog. A 90-day rolling window is a common default. For users with sparse history, you may need to extend the window or fall back to content-based signals. Always validate that extending the window actually improves offline metrics before paying the storage and compute cost.

Offline Evaluation

Don't just report AUC and call it done. AUC tells you the model can rank positives above negatives on average, but it hides a lot.

For retrieval, the metric you care about is Recall@K: of all items the user eventually interacted with, what fraction appear in your top-K candidates? If Recall@500 is 60%, you're leaving 40% of relevant items on the floor before ranking even runs. That's your ceiling.

For ranking, use NDCG@K (Normalized Discounted Cumulative Gain). It rewards putting the most relevant items at the top of the list, which maps directly to what the user sees. Also track MRR (Mean Reciprocal Rank) if you care about the first click specifically.

Time-based splits are non-negotiable. Never use random train/test splits on interaction data. If you do, you'll leak future interactions into training and inflate your metrics by 5-15 points. Split by time: train on interactions before date T, evaluate on interactions after T. This also tests whether your model generalizes to new items that weren't in the training catalog.

Common mistake: Candidates evaluate on a random 20% holdout and report great AUC. The interviewer asks "how did you prevent data leakage?" and the answer falls apart. Always use time-based splits.

Baseline comparisons should include at minimum: popularity-based ranking, the previous production model, and a simple collaborative filter. If your new model doesn't beat all three, you don't ship it.

Error analysis is where you separate good candidates from great ones. Break down your errors by user segment: new users (cold-start), power users, users in underrepresented regions. A model with 0.82 AUC overall might have 0.65 AUC on new users, which is a product disaster. Also look at item-side errors: are you systematically underranking new items? Long-tail content? Items in specific categories?

Calibration matters if your scores feed downstream systems. A model that outputs 0.9 for everything it likes isn't useful for threshold-based decisions. Check that your predicted CTR matches observed CTR across score buckets. If it doesn't, apply Platt scaling or isotonic regression as a post-processing step.

Tip: Interviewers want to see that you evaluate models rigorously before deploying. "It has good AUC" is not enough. Discuss calibration, fairness slices across user demographics, and failure modes like cold-start degradation and popularity bias. Mentioning these unprompted signals that you've shipped models to real users before.

Inference & Serving

Most recommendation systems need results in under 200ms, end-to-end, including network overhead. That's not a lot of time to fetch features, run retrieval, score hundreds of candidates, and apply business rules. Every millisecond you spend in one stage is a millisecond you can't spend in another.

Serving Architecture

Online vs. batch inference. For a homepage feed, you need online inference. The user's context (what they just watched, time of day, device) changes with every request, and a pre-computed batch recommendation would be stale before it's served. Batch inference makes sense for email digests or push notifications, where you can precompute recommendations for all users overnight and store them. For this system, the primary path is online.

The one exception worth mentioning: you can pre-compute and cache the retrieval stage output (the candidate set) for users who haven't had significant activity changes. This hybrid approach gives you the freshness of online ranking with the speed of pre-computed retrieval.

Model serving infrastructure. The two-tower retrieval model doesn't run at request time in the traditional sense. You pre-build the ANN index (FAISS or ScaNN) offline from all item embeddings, then at serving time you only run the user tower to get the query embedding and fire a nearest-neighbor lookup. That lookup is fast, typically under 20ms even for catalogs with 100M items.

The ranking model is where you need a proper model server. TFServing works well if you're in the TensorFlow ecosystem; Triton is better if you have mixed frameworks or want more control over batching and GPU memory management. Both support dynamic batching, which is essential for throughput.

# Triton model config for ranking model
name: "recommendation_ranker"
backend: "tensorflow"
max_batch_size: 256
dynamic_batching:
  preferred_batch_size: [64, 128]
  max_queue_delay_microseconds: 5000   # wait up to 5ms to fill a batch
input:
  - name: "user_features"
    data_type: TYPE_FP32
    dims: [-1, 128]
  - name: "item_features"
    data_type: TYPE_FP32
    dims: [-1, 64]
output:
  - name: "ranking_score"
    data_type: TYPE_FP32
    dims: [-1, 1]

Request flow end-to-end. Here's what actually happens when a user opens the app:

1. The API gateway receives the request with user ID and context (device, surface, timestamp).
2. The recommendation service fires two parallel calls: fetch user features from the online feature store (Redis, single-digit milliseconds), and run the user tower to get the user embedding.
3. The user embedding hits the FAISS index and returns the top-500 candidate items.
4. The recommendation service fetches item features for all 500 candidates from the feature store (batched lookup).
5. The ranking model scores all 500 candidates in a single batched inference call.
6. Post-processing applies business rules: diversity constraints, freshness boosts, deduplication against recently shown items.
7. The top 20-50 results are returned.

Latency breakdown. With a 200ms p99 budget, a realistic allocation looks like this:

That 60ms buffer matters. You'll exceed your targets on cold cache misses, slow feature store lookups, and garbage collection pauses. Design for the median, but instrument every stage so you know exactly where p99 latency is coming from.

Interview tip: Interviewers love asking "where does latency come from?" Walk through the stages explicitly and show you've thought about parallelism. Feature fetching and user tower inference can often run in parallel, shaving 10-15ms off the critical path.

---

Optimization

Model optimization. The ranking model is your biggest latency risk. INT8 quantization is usually the first thing to try; it cuts memory bandwidth and speeds up inference by 2-4x with minimal accuracy loss on ranking tasks. Run a calibration pass on representative data before deploying a quantized model, and verify your offline metrics don't regress more than your acceptable threshold (typically <0.5% AUC drop).

Knowledge distillation is worth considering if you're running a large transformer-based ranker. Train a smaller student model to mimic the larger teacher's output distributions. You often get 80% of the quality at 30% of the latency. Pruning is less commonly used in production recommendation systems because the structured sparsity gains are harder to realize on modern GPU hardware.

Prediction caching is underrated. If the same user makes two requests within a short window (say, 30 seconds) without any new interactions, you can return the cached result. Cache hit rates of 10-20% are common on mobile apps where users navigate back and forth. Just make sure your cache key includes enough context (surface, device type) to avoid serving a mobile-formatted feed to a desktop user.

Batching strategies. Dynamic batching is the key lever for GPU throughput. Without batching, each request occupies the GPU for a full forward pass with a batch size of 1, which is extremely wasteful. With dynamic batching, the model server waits up to a configurable timeout (5-10ms) to accumulate requests, then processes them together. At 1000 QPS, you'll naturally form batches of 50-100, which is where GPU utilization becomes efficient.

The tradeoff is latency variance. A request that arrives just after a batch dispatches has to wait for the next batch window. Set your max_queue_delay based on your latency budget, not your throughput goals.

GPU vs. CPU serving. The two-tower retrieval (user tower only) and the FAISS lookup are fast enough to run on CPU. The ranking model, especially if it's a deep network with attention layers, benefits significantly from GPU. A practical setup: run retrieval on CPU instances (cheaper, easier to scale horizontally) and ranking on GPU instances (more expensive, but you need far fewer of them due to batching efficiency).

For smaller ranking models (LightGBM or a shallow MLP), CPU serving is often fine and dramatically simpler to operate. Don't default to GPU just because it sounds more impressive.

Fallback strategies. Your model will be unavailable sometimes. Deployments, hardware failures, and cascading timeouts all happen. You need a fallback that's always available and returns in under 10ms. Good options: a pre-computed popularity-based ranking stored in Redis, a simple rule-based filter (trending in the user's preferred categories), or a cached version of the user's last successful recommendation response. Whatever you choose, make sure it degrades gracefully rather than returning an error to the user.

Common mistake: Candidates describe the happy path in detail but skip fallbacks entirely. Interviewers at Netflix, Spotify, and Meta will ask "what happens if the ranking service is down?" Have an answer.

---

Online Evaluation & A/B Testing

Offline metrics (AUC, NDCG) tell you if your model is better in a vacuum. Online metrics tell you if it's actually better for users. The gap between the two is real, and you can't skip A/B testing.

Traffic splitting. Assign users to experiment buckets deterministically using a hash of their user ID and the experiment ID. This ensures the same user always sees the same variant across sessions, which is critical for measuring cumulative effects like watch time or retention. Avoid random per-request assignment; it creates inconsistent experiences and makes it harder to detect effects that compound over time.

Run experiments at the user level, not the request level. If user A sees the new model on Monday and the old model on Tuesday, their behavior in both conditions is contaminated.

Metrics and statistical methodology. The primary metric for most recommendation experiments is engagement: CTR, watch time, or session length depending on the product. But always track guardrail metrics too: unsubscribe rate, skip rate, and explicit negative feedback. A model that increases CTR by 2% but doubles the skip rate is not a win.

Use a two-sample t-test or Mann-Whitney U test for continuous metrics. For ratio metrics (CTR), use a z-test on proportions or a bootstrap. Run your power analysis before the experiment starts to know how long you need to run it. Peeking at results and stopping early is one of the most common ways to get false positives.

Key insight: The deployment pipeline is where most ML projects fail in practice. A model that can't be safely deployed and rolled back is a model that won't ship.

Ramp-up and rollback. Start new models at 1-5% traffic. Watch your guardrail metrics for 24-48 hours before expanding. Most regressions are visible within the first few hours at scale. If CTR drops more than 2% relative to control, or if any guardrail metric moves in the wrong direction with statistical significance, roll back automatically.

Automated rollback requires you to define your thresholds before the experiment starts. If you define them after you see the data, you're p-hacking.

Interleaving experiments. For ranking systems specifically, interleaving is worth knowing about. Instead of showing user A only model A's results and user B only model B's results, you interleave both models' ranked lists for the same user in the same session and measure which model's items get clicked more. This dramatically reduces the variance of your experiment because each user serves as their own control. You can reach statistical significance with 10-100x fewer users than a standard A/B test. Netflix and LinkedIn have published extensively on this approach. It's a strong signal that you've thought deeply about recommendation-specific evaluation.

---

Deployment Pipeline

Validation gates. Before any model touches production traffic, it has to pass a set of automated checks. At minimum: offline metric regression (new model's NDCG@10 must be within X% of the current production model), prediction distribution sanity checks (score distributions shouldn't shift dramatically), and latency benchmarks (p99 inference latency under load must stay within budget). These gates run as part of your CI/CD pipeline and block promotion if they fail.

Don't skip the latency benchmark. A model that's 5% better on NDCG but 40% slower will blow your SLA in production.

Shadow scoring. Before canary, run the new model in shadow mode. It receives a copy of every production request, generates predictions, and logs them, but those predictions are never shown to users. This lets you verify that the model runs correctly at production traffic levels, that feature fetching works, that there are no serialization errors, and that latency is acceptable. Shadow mode catches infrastructure bugs that offline testing misses.

Canary rollout. Start at 1% of traffic. Hold for at least one full day-night cycle, because user behavior varies significantly by time of day. Expand to 5%, then 10%, then 25%, then 50%, then 100%, with monitoring checkpoints at each stage. Each expansion should be gated on the previous stage's metrics being clean.

# Simplified canary rollout schedule
ROLLOUT_STAGES = [
    {"traffic_pct": 1,   "hold_hours": 24,  "auto_advance": True},
    {"traffic_pct": 5,   "hold_hours": 24,  "auto_advance": True},
    {"traffic_pct": 10,  "hold_hours": 48,  "auto_advance": False},  # manual review
    {"traffic_pct": 25,  "hold_hours": 48,  "auto_advance": True},
    {"traffic_pct": 50,  "hold_hours": 24,  "auto_advance": True},
    {"traffic_pct": 100, "hold_hours": 0,   "auto_advance": False},
]

ROLLBACK_TRIGGERS = {
    "ctr_relative_drop": 0.02,          # 2% relative drop vs control
    "p99_latency_ms_threshold": 180,    # absolute latency threshold
    "error_rate_threshold": 0.001,      # 0.1% error rate
}

Rollback triggers. Rollback should be one button push, or better yet, fully automated. The rollout controller monitors your guardrail metrics continuously. If any trigger fires, it immediately shifts 100% of traffic back to the previous model version without waiting for human intervention. The previous model artifact stays warm in the serving infrastructure specifically for this reason. Blue-green deployments make this clean: the old model is still running, you just flip the traffic routing.

Interview tip: When you describe your deployment pipeline, mention that the previous model stays warm. Interviewers notice when candidates think about the operational details, not just the happy path.

Monitoring & Iteration

Most candidates treat deployment as the finish line. It isn't. A recommendation model that isn't actively monitored will quietly degrade, and in a system with feedback loops, it can actively make things worse over time.

Tip: Staff-level candidates distinguish themselves by discussing how the system improves over time, not just how it works at launch.

Production Monitoring

Think of monitoring in three layers. Each catches different failure modes, and you need all three.

Data monitoring is your first line of defense. Before the model even runs, check that the inputs look sane. Feature drift happens constantly in recommendation systems: user behavior shifts seasonally, item catalogs change, upstream pipelines break silently. Track the distribution of key features (watch time, click rates, user activity buckets) using Population Stability Index (PSI) or KL divergence. Set alerts when PSI exceeds 0.2 on any high-importance feature. Schema violations and missing feature rates should be near-zero; if a feature that was 99% populated drops to 60%, your model is running on garbage.

Model monitoring is where most teams underinvest. Watch the distribution of your model's output scores, not just aggregate metrics. If your ranker used to output scores between 0.1 and 0.9 and suddenly everything is clustering near 0.3, something upstream changed. Embedding drift is subtler: if your two-tower model's user embeddings start shifting in cosine distance from their historical positions, your retrieval quality is degrading even if no code changed. Tools like Evidently AI or custom Prometheus dashboards work here.

System monitoring is table stakes but still gets missed. Track p50/p95/p99 latency at each stage of the serving path separately: feature fetch, ANN retrieval, ranking inference. A latency spike in the feature store looks very different from a GPU saturation issue in the ranking service. GPU utilization, batch queue depth, and model server error rates should all have dashboards and paging thresholds.

For alerting, tier your thresholds. A 5% drop in CTR over 24 hours is a Slack notification. A 20% drop in 1 hour is a page. Schema violations on critical features should page immediately, because the model is already serving bad results.

Common mistake: Candidates set up monitoring on offline metrics only. If your NDCG looks fine in batch evaluation but live CTR is dropping, you have a serving skew problem that offline monitoring will never catch.

Feedback Loops

User interactions are your ground truth, but they arrive messy and delayed.

The basic loop: a user sees a recommendation, clicks or ignores it, and that signal flows back through your event pipeline into training data. Clicks and watch-time arrive within seconds. Conversions (purchases, saves, subscriptions) might take hours. Ratings, if you collect them, might take days or never arrive at all.

Feedback delay is a real design problem. If you retrain daily on yesterday's data, you're always training on a slightly stale label distribution. For high-churn catalogs (news, live events), this matters a lot. One pattern: use immediate implicit signals (clicks, session depth) as your fast-feedback training signal, and treat slower signals (purchases, ratings) as a secondary fine-tuning signal on a longer cadence. Don't wait for perfect labels when good-enough labels are available now.

The more dangerous feedback loop is the popularity spiral. A model learns that popular items get clicked. It recommends popular items more. Those items get more clicks. The model doubles down. Within weeks, your long-tail catalog is invisible, your diversity metrics tank, and users start complaining that everything looks the same. You need to actively monitor catalog coverage (what fraction of your item catalog appears in recommendations at all) and inject diversity constraints at re-ranking time before this spiral takes hold.

Key insight: Feedback loops don't just affect model quality. They affect what data you collect next. A biased model creates biased training data, which trains a more biased model. Breaking this cycle requires intentional exploration, not just better optimization.

Closing the loop from alert to fix follows a standard path: monitoring fires an alert, on-call engineer diagnoses whether it's data drift, model degradation, or a serving bug, and then either triggers an emergency retrain or rolls back to the previous model version. Having a clean rollback path is non-negotiable. Your model registry (MLflow, Weights & Biases) should store every promoted artifact with its evaluation metrics so you can redeploy a previous version in minutes.

Continuous Improvement

Scheduled vs. triggered retraining is a question of how fast your data distribution moves. A music recommendation system serving a stable catalog can retrain weekly without much harm. A news feed recommendation system where the item catalog turns over hourly needs daily or even continuous online learning. The practical answer for most teams: start with scheduled daily retraining, instrument drift detection, and use drift alerts to trigger out-of-cycle retrains when the distribution shifts suddenly (a major news event, a product launch, a viral moment).

Full retraining from scratch vs. incremental fine-tuning is a cost/quality tradeoff. Full retraining is more stable and avoids catastrophic forgetting, but it's expensive and slow. Fine-tuning on recent data is cheap and fast, but can overfit to short-term trends. Most mature systems do both: daily fine-tuning on recent interactions, weekly full retrains to recalibrate.

Prioritizing improvements is where engineering judgment matters. The hierarchy is roughly: fix data quality issues first (they compound everything downstream), then add high-signal features (session context, real-time signals), then improve model architecture. A cleaner training dataset almost always beats a fancier model on the same dirty data.

As the system matures, the problems shift. Early on, you're fighting cold start and data sparsity. At scale, you're fighting staleness, feedback loop bias, and the operational cost of keeping everything running. The offline-online metric gap becomes a persistent headache: a model with better NDCG offline sometimes performs worse online because it was evaluated on a different item distribution than it serves. Interleaving experiments (where two models compete for the same user's feed in real time) and counterfactual evaluation using logged propensity scores are the right tools for closing that gap, and knowing to reach for them is a signal of senior-to-staff level thinking.

Common mistake: Optimizing a single metric (CTR) in isolation. At scale, you need to jointly track engagement, diversity, and long-tail coverage. A model that maximizes CTR by serving only blockbuster content is a business risk, not a success.

The systems that age well are the ones built with iteration in mind from day one: clean feature pipelines that prevent training-serving skew, a model registry with rollback capability, drift detection that triggers retraining automatically, and monitoring that covers data, model, and business health simultaneously. That's the difference between a recommendation system and a recommendation system you can actually maintain.

What is Expected at Each Level

Interviewers at Netflix, Meta, and Spotify aren't grading you on a single rubric. They're calibrating your level. The same question gets a different passing bar depending on the role you're interviewing for, and knowing where that bar sits is half the battle.

Mid-Level

• Design the two-stage pipeline. You should be able to explain why retrieval and ranking are separate stages, name a reasonable model for each (two-tower for retrieval, LightGBM or wide-and-deep for ranking), and articulate why you can't just run the ranking model over the entire catalog.
• Identify the right features. User history, item metadata, recency signals, and session context. You don't need to enumerate every feature, but you should demonstrate that you know the difference between user-level, item-level, and interaction-level signals.
• Handle cold-start at a high level. New users get popularity-based or demographic-based recommendations. New items get content-based embeddings until they accumulate interaction data. You don't need a complete solution, just show you've thought about it.
• Sketch a basic A/B testing plan. Traffic splitting, a control group, and a primary metric. Bonus points for mentioning that you'd run the experiment long enough to account for novelty effects.

Common mistake: Mid-level candidates often jump straight to model architecture without defining what they're optimizing for. State your ML objective before you touch model design.

Senior

• Catch training-serving skew before the interviewer brings it up. Explain that your feature transformations need to run identically in the training pipeline and at serving time, and that a shared feature store with versioned transforms is how you enforce that. This is the detail that separates candidates who've shipped recommendation systems from those who've only read about them.
• Go deep on negative sampling. Random negatives are easy to implement but produce overconfident models. In-batch negatives are more realistic. Popularity-weighted sampling corrects for exposure bias. You should be able to explain the tradeoff and pick one with justification.
• Reason about the latency budget. The p99 target is 200ms end-to-end. Feature fetch takes ~10ms, ANN retrieval takes ~20ms, ranking model inference takes ~50ms. You should be able to sketch where the time goes and identify which stage is the bottleneck under load.
• Propose a concrete monitoring plan. Not just "we'll watch CTR." Specify what you're measuring at the data layer (feature drift, missing values), the model layer (score distribution shift), and the business layer (engagement, diversity). Name a retraining trigger condition.

Key insight: Senior candidates don't just design the happy path. They anticipate what breaks in production and build the system to catch it.

Staff+

• Reason about real constraints, not ideal ones. A catalog of 100M items with 50K QPS and a two-person ML team has different answers than a catalog of 10K items at 1K QPS. Staff candidates size the system to the actual problem, make explicit tradeoffs between model complexity and operational cost, and know when a simpler model that the team can maintain beats a sophisticated one that becomes a black box.
• Design the experimentation platform, not just the experiment. That means thinking about how to run five simultaneous A/B tests without interference, how to handle novelty bias in short-run experiments, and when interleaving is a better evaluation method than a traditional holdout split.
• Address long-term ecosystem health. A recommendation system that only optimizes CTR will eventually collapse into a popularity feedback loop, starving long-tail content and eroding catalog diversity. Staff candidates bring this up unprompted and propose concrete mechanisms: diversity constraints in re-ranking, exploration budgets, or fairness-aware training objectives.
• Map the cross-team dependencies. The feature store is owned by data engineering. The A/B testing platform is owned by growth. The item catalog is owned by content. Staff candidates identify these dependencies, flag the coordination risks, and propose interfaces that let the ML team move independently without breaking shared infrastructure.

Key takeaway: The difference between a good recommendation system and a great one isn't the model architecture. It's whether the system can learn, adapt, and stay healthy over months and years without requiring heroic intervention every time the data distribution shifts.