NLP Interview Questions

Sample Answer

Skip-gram is the better choice for corpora with many rare words. Skip-gram predicts context words given a target word, meaning each occurrence of a rare word generates multiple training examples (one per context word), giving rare tokens more gradient updates. CBOW averages context vectors to predict the center word, which works well for frequent words but tends to smooth over rare terms since they appear infrequently as targets. In practice, Skip-gram with negative sampling is the standard approach when you care about the quality of representations for the long tail of your vocabulary.

Sample Answer

You could use off-the-shelf GloVe embeddings or train Word2Vec on your domain-specific corpus. Training Word2Vec on your internal data wins here because content moderation text is full of slang, intentional misspellings, and coded language that GloVe (trained on Wikipedia or Common Crawl) will either map to generic vectors or miss entirely as out-of-vocabulary tokens. By training on your own data, you capture the distributional semantics of adversarial terms in context, so "unalive" and similar euphemisms land near their true meanings. A practical middle ground is to initialize with GloVe and fine-tune on your domain corpus, but if your internal data is large enough, training from scratch with Word2Vec or FastText (which handles misspellings via subword information) is more robust.

Sample Answer

You could fine-tune BERT for maximum accuracy or use TF-IDF plus logistic regression for speed. Under a strict 10ms latency constraint, the TF-IDF plus logistic regression pipeline wins because inference is sub-millisecond, while even a distilled BERT model typically requires 5 to 20ms on CPU. A strong middle ground is to fine-tune a small transformer like DistilBERT, then apply ONNX optimization and quantization to push inference under 10ms. If accuracy on the 50 intent classes is critical and you have GPU serving infrastructure, you can serve the fine-tuned transformer with batched inference, but you should benchmark end-to-end latency including tokenization before committing.

Sample Answer

First, you want to slice your evaluation set by linguistic phenomena: create subsets for sarcastic comments, code-switched text, slang-heavy posts, and standard language, then compute per-slice precision, recall, and F1 to pinpoint exactly where performance degrades. Next, you should check whether your training data adequately represents these phenomena. If sarcasm and code-switching are underrepresented, you need targeted data collection or augmentation, for example by mining sarcasm-tagged subreddits or multilingual social media corpora. Then consider whether your model architecture captures enough context: sarcasm often requires understanding pragmatic cues, so a larger pretrained model fine-tuned on diverse social media text (like XLM-R for code-switching) will outperform models trained only on formal English. Finally, you can add auxiliary signals such as emoji usage patterns or user-level features, and implement a human-in-the-loop review pipeline for low-confidence predictions in these difficult slices.

Sample Answer

First, you need a chunking strategy: you split the document into overlapping windows, typically with a stride of 128 or 256 tokens, so that most entities appear fully within at least one chunk. Next, you run inference on each chunk independently, then merge predictions in the overlapping regions by keeping the prediction from the chunk where the token is furthest from the boundary, since edge tokens tend to have weaker contextual representations. For training, you assign labels only to the non-overlapping core of each chunk to avoid double-counting loss on shared tokens. If you still encounter split entities at boundaries, a post-processing step that joins compatible B/I tags across adjacent chunks using simple heuristics or a lightweight sequence model resolves most remaining errors.

Sample Answer

This question is checking whether you can articulate the vanishing gradient problem and connect it concretely to the LSTM gating mechanism. In vanilla RNNs, gradients are multiplied by the recurrent weight matrix at each timestep during backpropagation, causing them to shrink exponentially and making it nearly impossible to learn dependencies between distant tokens. LSTMs solve this with a cell state $c_t$ that flows through time with additive updates: $c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t$, where the forget gate $f_t$ and input gate $i_t$ control what information persists or enters. This additive path creates a gradient highway that avoids repeated multiplication, letting the model remember that a token 30 positions back was the start of an entity.

Sample Answer

This question is checking whether you can connect architectural choices to concrete task requirements, not just recite definitions. Encoder-only models like BERT use masked language modeling (MLM), where you mask random tokens and predict them using bidirectional context, making them strong for classification and extraction tasks. Decoder-only models like GPT use causal language modeling, predicting the next token left to right, which naturally suits open-ended generation. Encoder-decoder models like T5 combine both: the encoder processes the full input bidirectionally, and the decoder generates output autoregressively, which is ideal for sequence-to-sequence tasks like translation or summarization. You should pick decoder-only when generation quality and simplicity of scaling are priorities, encoder-only for discriminative tasks, and encoder-decoder when you have a clear input-output mapping with differing lengths.

Sample Answer

The standard move is to use RoPE or learned positional embeddings, since they tend to yield better downstream performance and, in RoPE's case, offer elegant length extrapolation by encoding relative positions directly into the attention dot product via rotation matrices. But sinusoidal encodings still matter because they require zero additional parameters, generalize deterministically to unseen sequence lengths, and are useful in resource-constrained or fixed-architecture settings where you want a simple, well-understood baseline. Learned embeddings are limited to the maximum length seen during training unless you add interpolation tricks. You should mention that RoPE has become the default in models like LLaMA precisely because it encodes relative position without extra memory cost while maintaining strong extrapolation, making it the best of both worlds for most production LLMs.

Sample Answer

Get this wrong in production and you waste weeks of GPU compute on a model that never learns meaningful token dependencies. Near-uniform attention distributions suggest the attention logits are too small to differentiate between positions, which often points to poor initialization, a learning rate that is too low for the attention parameters, or numerical issues where the $QK^T$ products are collapsing in magnitude. You should check the variance of $Q$ and $K$ projections at initialization to ensure they produce dot products with reasonable scale, verify that your $\sqrt{d_k}$ scaling is correctly applied, and inspect whether layer normalization or weight initialization (such as using too-small fan-in scaling) is suppressing signal. Fixes include adjusting initialization to match the expected variance, tuning the learning rate with proper warmup, and potentially using QK-normalization to stabilize attention score magnitudes.

Sample Answer

The standard move is to use beam search for tasks with a narrow set of correct outputs, like machine translation or summarization. But here, open-ended generation matters because beam search maximizes likelihood, which empirically leads to bland, repetitive, and degenerate text, a problem that gets worse as beam width increases. A width of 50 amplifies this by concentrating probability mass on high-frequency token sequences. You should recommend stochastic decoding instead, such as top-p or top-k sampling combined with a moderate temperature like $T = 0.7$ to $1.0$, which produces diverse and natural-sounding stories.

Sample Answer

Get this wrong in production and your model becomes a glorified lookup table that collapses to greedy decoding, losing all ability to express uncertainty or produce diverse responses. Temperature scaling modifies the softmax distribution by dividing logits by $T$ before normalization: $$p_i = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}$$. As $T \to 0$, the distribution becomes a point mass on the highest-logit token, which causes repetitive outputs, inability to recover from early mistakes in autoregressive generation, and poor performance on tasks like brainstorming or creative writing where multiple valid continuations exist. You should explain to the PM that the right temperature depends on the use case: near-zero for factual QA, higher values like $0.7$ to $1.0$ for open-ended tasks.

Sample Answer

Get this wrong in production and you tank user satisfaction and retention because people abandon a model that refuses to help with legitimate queries. The right call is to first build a taxonomy of refusal types by sampling refused completions and labeling them as true positives (correctly refused harmful content) versus false positives (benign requests incorrectly refused). Then you retrain or adjust your reward model with additional preference data that explicitly ranks helpful compliance above unnecessary hedging for safe prompts, effectively recalibrating the boundary. You can also apply a targeted round of PPO or DPO with curated prompts from the false-positive category, using a KL penalty coefficient $\beta$ tuned to prevent the model from swinging too far toward compliance on genuinely harmful inputs. Monitoring the false refusal rate alongside safety metrics post-deployment closes the loop.

Sample Answer

Naively quantizing to INT4 sounds reasonable but breaks under long-form summarization where accumulated quantization error degrades coherence. Reducing max output tokens alone doesn't work because the task inherently requires multi-sentence summaries. That leaves a combination of speculative decoding (using a small draft model to propose tokens verified in parallel by the large model), INT8 or FP8 weight-only quantization (which preserves quality far better than INT4 for generative tasks), and KV-cache optimization techniques like PagedAttention via vLLM to maximize throughput. You should also consider distilling the 13B model into a 3B-7B student model fine-tuned specifically on summarization, which can hit sub-1-second latency natively. Benchmark each approach on your eval set with ROUGE and human preference scores to confirm quality holds.