LLMs & Transformers Interview Questions

Sample Answer

You should choose pre-norm for a 70B parameter model. Pre-norm places LayerNorm before the attention and FFN sub-layers, which keeps the residual stream as a clean identity path and makes gradient magnitudes more uniform across depth. This dramatically improves training stability at scale, often eliminating the need for careful learning rate warmup that post-norm requires. Post-norm can yield slightly better final performance in smaller models, but at 70B parameters the training instability risks make it impractical without significant additional engineering.

Sample Answer

You could use sinusoidal positional encodings added to token embeddings, or you could use RoPE which applies rotation matrices directly to query and key vectors. RoPE wins here because it encodes relative position directly into the attention dot product: the inner product $\text{Re}[(R_\theta^m q)(R_\theta^n k)^*]$ depends only on $m - n$, giving the model an inductive bias toward relative distance without extra parameters. Sinusoidal encodings are absolute and do not naturally generalize beyond trained lengths, whereas RoPE, combined with techniques like NTK-aware interpolation, extends gracefully to 32K tokens and beyond. This is why nearly every modern open-weight LLM, including LLaMA and Mistral, uses RoPE.

Sample Answer

Let's reason through this step by step. A single head computes one set of attention weights per token pair, meaning it can only capture one interaction pattern per layer. By splitting into $h$ heads, each head learns its own $W_Q^i, W_K^i, W_V^i$ projections operating in a $d_k$-dimensional subspace, so different heads can attend to different positions for different reasons simultaneously, such as one head tracking syntactic dependencies while another tracks coreference. The total compute is the same as a single large head since $h \times d_k = d_{\text{model}}$, but the representational diversity is far greater. Empirically, ablations in the original transformer paper and subsequent work confirm that multiple heads consistently outperform a single head of equal total dimension.

Sample Answer

You could use BPE, which greedily merges the most frequent adjacent pairs, or the SentencePiece unigram model, which starts with a large vocabulary and iteratively prunes tokens by minimizing the corpus likelihood under a unigram language model. The unigram approach wins when you need probabilistic tokenization with multiple segmentation candidates, because it naturally supports subword regularization during training, which acts as data augmentation and improves robustness. BPE is simpler, deterministic, and widely battle-tested in production, making it easier to debug and reproduce. SentencePiece also operates directly on raw Unicode without pre-tokenization rules, which gives it an edge for languages without clear whitespace boundaries like Japanese or Thai.

Sample Answer

Let's reason through this step by step. When BPE tokenizes a number like "12345," it might merge it into a single token or split it inconsistently, say "123" and "45," meaning the model never learns a stable positional notion of individual digits. If the model sees "1" as part of different multi-character tokens depending on context, it cannot reliably learn place value, which is fundamental to arithmetic. One concrete fix is to force digit-level tokenization by adding a pre-tokenization rule that splits every digit into its own token, so "12345" always becomes ["1", "2", "3", "4", "5"]. This is exactly what models like LLaMA adopted, and you can also reverse the digit order (least significant first) to align the addition carry direction with the left-to-right generation order, further improving arithmetic accuracy.

Sample Answer

Let's walk through this step by step. First, you supervised fine-tune (SFT) the base model on high-quality demonstrations. Second, you train a reward model on human preference comparisons, where annotators rank model outputs. Third, you optimize the SFT model against that reward model using PPO with a KL penalty $\beta \cdot D_{KL}(\pi_\theta \| \pi_{\text{ref}})$ to keep the policy close to the reference. Reward hacking happens when the policy exploits spurious patterns in the reward model, like generating verbose or repetitive outputs that score high but are low quality, meaning your reward model has poor out-of-distribution generalization. You fix this by increasing the KL penalty coefficient $\beta$, retraining the reward model on adversarial examples from the current policy (iterated RLHF), or switching to direct preference optimization (DPO) which sidesteps the explicit reward model entirely.

Sample Answer

This question is checking whether you can connect pre-training objectives to downstream task requirements rather than just reciting definitions. Masked language modeling (MLM) gives BERT bidirectional context, which is powerful for understanding tasks like classification and extraction, but it cannot generate text autoregressively. Causal language modeling (CLM) trains the model to predict the next token given all previous tokens, which directly matches the generation pattern needed for code completion. For code generation you would choose CLM because code is written left to right, functions depend on prior definitions, and you need the model to produce coherent continuations token by token at inference time.

Sample Answer

The standard move is to mix a fraction of general-purpose pre-training data (or a broad instruction dataset like FLAN) into your fine-tuning data to prevent catastrophic forgetting. But here, the severity of the drop also matters because if MMLU degrades by more than a few points, you should check whether your instruction dataset is too narrow or too small, which causes the model to overfit to a restricted output distribution. Reducing the learning rate, shortening training, or using LoRA instead of full fine-tuning can all limit forgetting since fewer parameters change. A practical ratio is 10 to 20 percent general data mixed with your task-specific instructions, tuned by monitoring both your target metric and held-out general benchmarks throughout training.

Sample Answer

This question is checking whether you can reason about loss curve dynamics beyond the textbook monotonic decrease. The plateau followed by a sudden drop is often associated with a phase transition where the model is internally reorganizing representations before a capability emerges, sometimes called a "breakthrough" or grokking-like phenomenon. You should distinguish this from learning rate schedule effects (check if a warmup restart or decay step coincides with the drop), data distribution shifts in the shuffling order, or batch size changes. To diagnose, you would inspect gradient norms, activation statistics, and per-task loss decompositions across the plateau to see if internal restructuring is happening even while aggregate loss is flat. Mentioning that emergent abilities in large models may manifest as exactly this kind of discontinuous loss behavior shows deeper understanding.

Sample Answer

The standard move is to follow Chinchilla-optimal allocation, which for this compute budget would likely favor something closer to the 7B/1T split. But here, inference cost matters because in production you serve the model millions of times, so a smaller 3B model with more training data (the LLaMA/Mistral philosophy of "over-training" relative to Chinchilla) can yield a better cost-performance tradeoff at deployment. You should frame this as: Chinchilla optimality minimizes loss per training FLOP, but total cost of ownership includes inference, so training a smaller model longer than compute-optimal can be the right business decision. The 3B model at 2.3T tokens will have higher training loss than the 7B at 1T, but the gap may be small and the inference savings enormous.

Sample Answer

The standard move is to use a small draft model to propose $K$ tokens in parallel, then verify them all in a single forward pass of the large target model, accepting tokens that match the target distribution. But here, the acceptance rate is what determines the actual speedup, because if the draft model is poorly aligned with the target model's distribution, most speculated tokens get rejected and you waste the draft computation. Specifically, the expected tokens accepted per step is $\sum_{i=1}^{K} \prod_{j=1}^{i} \alpha_j$ where $\alpha_j$ is the acceptance probability at position $j$, so low agreement compounds quickly. Speculative decoding also fails to help when the bottleneck is already compute-bound rather than latency-bound, such as in large-batch throughput-oriented serving, since verification of $K$ tokens costs nearly the same as generating $K$ tokens independently in a batched setting.

Sample Answer

Get this wrong in production and your model silently degrades on the hardest queries while your offline evals look fine. The issue is that GPTQ calibrates quantization using a small calibration set, and if that set does not represent long-context reasoning patterns, the layer-wise Hessian approximation underweights the channels critical for multi-hop attention over long sequences. Perplexity on short benchmarks is an average metric that hides tail degradation. You should try recalibrating with a representative long-context dataset, switching to AWQ which preserves salient weight channels based on activation magnitudes rather than second-order weight statistics, or using a mixed-precision approach where attention projection layers stay in FP16 while feedforward layers are quantized to INT4.

Sample Answer

Get this wrong in production and you ship a model whose capabilities are fundamentally misrepresented, leading to failures on real user queries that the benchmark scores said it could handle. The right call is to run a contamination audit: check for near-exact n-gram overlap between benchmark examples and the training corpus, test on rephrased or perturbed versions of benchmark items to see if performance drops sharply (which signals memorization over generalization), and compare performance on the canonical split versus held-out items from the same distribution that were never publicly released. You should also use canary string detection or membership inference techniques if you have access to model internals. If contamination is confirmed, you report results on the clean subset only, create a decontaminated evaluation split, and consider adopting dynamic benchmarks that rotate examples over time.

Sample Answer

Saying Constitutional AI removes humans entirely sounds reasonable but breaks under scrutiny, because humans still write the constitution (the set of principles) and design the critique/revision prompts. Arguing it is strictly better than RLHF also does not work because the AI feedback model can inherit or amplify blind spots present in its own training. That leaves the accurate framing: Constitutional AI replaces human preference labeling in the RLHF loop with AI-generated critiques and revisions guided by explicit principles, then trains a preference model (RLAIF) on the AI-ranked outputs. It scales better and reduces labeler subjectivity, but it can fail when the constitution is underspecified for novel harm categories, when the critique model lacks the capability to recognize subtle issues like implicit bias or factual errors, or when the principles conflict and there is no clear resolution hierarchy.