Evaluation & Benchmarks Interview Questions

Sample Answer

The core reason is that static multiple-choice benchmarks are proxy metrics that fail to capture the open-ended, context-dependent nature of real user interactions. Benchmark contamination is one factor: the model may have seen test data during training, inflating scores without genuine capability. Beyond that, benchmarks test narrow, well-defined tasks while production queries involve ambiguity, multi-turn dialogue, instruction following nuance, and domain-specific knowledge. You should investigate by sampling production failures, categorizing them (e.g., hallucination, refusal, style mismatch), and building targeted evals that mirror those failure modes. Closing this gap requires continuous evaluation on live traffic distributions, not just static test sets.

Sample Answer

You could do full human evaluation or LLM-as-judge. LLM-as-judge wins here because the 48-hour window and 10,000-example scale make human annotation prohibitively slow and expensive. However, you should not blindly trust the LLM judge: first calibrate it by running both human raters and the LLM judge on a shared subset of 200 to 500 examples, measuring inter-rater agreement (e.g., Cohen's $\kappa$) to validate that the judge correlates well with human preferences. If agreement is high ($\kappa > 0.7$), deploy the LLM judge at scale with structured rubrics and randomized ordering to reduce positional bias. Reserve human evaluation for a stratified sample of edge cases or disagreements flagged by the automated judge.

Sample Answer

You could use BERTScore for fast, embedding-based semantic similarity or LLM-as-a-judge for richer, rubric-based evaluation. LLM-as-a-judge wins here when your responses are open-ended and quality depends on reasoning, tone, or multi-dimensional criteria that a single similarity score cannot capture. BERTScore is preferable when you have strong reference answers, need low-latency batch evaluation, and want deterministic scores without the cost and variance of repeated LLM calls. In practice, you might use BERTScore as a cheap filter and reserve LLM-as-a-judge for a stratified sample to calibrate and audit the pipeline.

Sample Answer

First, you need to recognize that these three metrics operate on different scales and measure different things: perplexity captures fluency, BERTScore captures semantic fidelity, and LLM-as-a-judge captures holistic quality. You would start by normalizing each metric to a common range, for example using min-max normalization based on a calibration set with known human preference rankings. Then you would learn the weights by fitting a simple model (logistic regression or even a linear combination) that predicts human preference labels from the three normalized scores. You should validate that the composite score's ranking correlation (Kendall's $\tau$ or Spearman's $\rho$) with human judgments exceeds any individual metric alone. Finally, monitor for drift in production by periodically sampling outputs and re-calibrating weights as your model or user expectations change.

Sample Answer

This question is checking whether you understand the assumptions behind BLEU and when they break down. BLEU computes n-gram precision against reference translations, which assumes there is a small set of correct outputs with predictable phrasing. Customer support responses are highly variable in valid wording, so BLEU will penalize perfectly good answers that use different vocabulary. You should recommend a semantic similarity metric like BERTScore for automated evaluation, combined with an LLM-as-a-judge setup that scores along dimensions like helpfulness, accuracy, and tone, calibrated against a labeled set of human preference judgments.

Sample Answer

First, you would check for near-duplicate overlap between the training corpus and GSM8K test examples using n-gram matching or embedding similarity, flagging any test problem that appears verbatim or with minor paraphrasing in the training data. Next, you would create a set of novel problems that are structurally isomorphic to GSM8K problems but use different numbers, names, and surface forms, then compare accuracy on originals versus these perturbed variants. A significant accuracy drop, say from 85% to 60%, is strong evidence of memorization rather than genuine reasoning. Finally, you could measure calibration: contaminated models tend to be overconfident on memorized examples, so you would look for a gap in predicted probability distributions between suspected contaminated and clean subsets.

Sample Answer

This question is checking whether you can identify the concrete limitations of an existing benchmark and propose principled improvements. HumanEval has only 164 problems, limited test case coverage per problem, and a narrow distribution skewed toward short self-contained functions, so you would want a larger and more diverse problem set spanning multi-file contexts, library usage, and edge-case-heavy specifications. You should also design for contamination resistance by including a private held-out split that rotates periodically and by generating problems procedurally so surface forms can be refreshed. Finally, you would move beyond pass@$k$ as the sole metric and incorporate partial credit scoring, efficiency measurement, and code quality signals to better reflect real engineering competence.

Sample Answer

This question is checking whether you can distinguish between noise from ambiguous guidelines versus noise from undertrained raters. You should first stratify agreement by label category and annotator to see if low $\kappa$ is driven by specific confusing categories or by a subset of raters who diverge systematically. If confusion matrices show that most disagreement clusters around two adjacent categories (e.g., "mildly unsafe" vs. "borderline"), the fix is sharper guidelines and possibly collapsing categories, not more labels. If instead certain annotators consistently deviate from the majority, targeted retraining or removal is more efficient. Only after tightening guidelines should you consider increasing redundancy, because adding labels per example with a broken taxonomy just averages over confusion.

Sample Answer

The standard move is to compute the correlation between response length and win rate across your dataset, then flag it if $r > 0.3$ or so. But here, the subtlety matters because some tasks genuinely require longer answers, so a blanket length penalty would hurt valid preferences. You should stratify by prompt type: if length predicts preference even within categories where concise answers are clearly better (e.g., factoid QA), you have confirmed bias. Mitigation options include normalizing response lengths in the pairs shown to annotators, adding explicit instructions to ignore length, or including "trap" pairs where the shorter response is objectively better to calibrate and filter biased raters.

Sample Answer

The standard move is to tighten the safety classifier's decision boundary or adjust system prompt instructions to be more permissive on borderline topics. But here, the key nuance is that you need a paired evaluation set: one with genuinely harmful prompts in the same domain (e.g., synthesis of dangerous compounds) and one with benign analogs (e.g., undergraduate chemistry questions). You measure both the false positive rate (over-refusal on benign) and the true positive rate (correct refusal on harmful) simultaneously, treating it as a precision-recall tradeoff. Adjust thresholds or fine-tune with contrastive examples until you hit an acceptable operating point, and track the metric $F_\beta$ with $\beta < 1$ if you want to weight precision over recall for the safety classifier.

Sample Answer

Get this wrong in production and you ship a model that looks safe on dashboards but leaks harmful content through soft compliance, partial answers, or encoded outputs. The right call is to measure actual harm in the response content, not just the presence of a refusal string. You should use a graded rubric (e.g., full refusal, partial compliance, full compliance) scored by a calibrated LLM judge or human annotators, and separately track whether the refusal is appropriate (true positive) versus unnecessary (false positive). This gives you a 2x3 confusion matrix across harm categories that tells a much richer story than a single refusal count.

Sample Answer

Get this wrong in production and you either ship a worse prompt that degrades user trust or you run the test for months without reaching significance because your variance is too high. The right call is to use a combination of automated LLM-as-judge scoring on dimensions like helpfulness, coherence, and safety, paired with implicit user signals such as conversation continuation rate, thumbs up/down ratio, and task completion rate. You should randomize at the user level, not the request level, to avoid within-session inconsistency. To handle high output variance, pre-register your primary metric, use stratified sampling by query type, and consider paired comparisons where both prompts generate responses for the same input and a judge model picks the better one. Target a minimum detectable effect size that is business-meaningful, typically around 2 to 5 percent relative improvement on your primary metric, and run power analysis accordingly.

Sample Answer

Running a simple accuracy check on a static test set sounds reasonable but breaks under distribution shift, since your golden set goes stale within weeks. A single aggregate metric does not work because it masks category-level regressions that affect specific user segments. That leaves you with a layered pipeline: first, sample recent production queries and run them through both retrieval quality checks (recall@k, chunk relevance scores) and generation quality checks (faithfulness, answer relevance via LLM-as-judge). You set two threshold tiers: a warning threshold based on a rolling 7-day baseline with $z$-score $> 2$ that pages the on-call team for investigation, and a critical threshold at $z$-score $> 3$ or any safety metric violation that triggers automatic rollback to the last known good configuration. Include slice-level breakdowns by query category so a 15 percent drop in medical queries does not get hidden by stable performance elsewhere.