RAG & Vector Databases Interview Questions

Sample Answer

You use a two-stage retrieval pipeline: a cheap, GPU-free sparse retriever like BM25 as the first pass over all 10 million documents, followed by a lightweight cross-encoder reranker on only the top 50 to 100 candidates. This lets you avoid embedding and indexing all 10 million documents with a dense model upfront, which saves significant GPU cost. For generation, you can call a smaller fine-tuned model or use a hosted API with a well-constructed prompt containing only the top 5 reranked chunks. If accuracy still falls short, you invest GPU budget selectively into dense embeddings for the highest-traffic document collections rather than the entire corpus.

Sample Answer

You could skip reranking and stuff all retrieved chunks into the prompt, or you could add a reranker to filter and order them before augmentation. Reranking wins here in most production scenarios because naively stuffing chunks introduces noise that degrades answer quality, increases token cost proportional to the number of chunks, and can push relevant context outside the model's effective attention window. The only case where skipping reranking makes sense is when you retrieve very few chunks (say 3 to 5) from a high-precision index and your context window is large relative to the content, making the marginal cost of a reranker not worth the added latency. In practice, a cross-encoder reranker on 20 to 50 candidates narrowed down to 3 to 5 chunks gives you the best accuracy-to-cost ratio.

Sample Answer

Start by recognizing that a single embedding lookup captures only one semantic facet of a multi-hop question, so you need to decompose the query. First, you use the LLM to break the complex question into sub-questions, each targeting a distinct information need. Then you run retrieval independently for each sub-question, gathering chunks from potentially different case documents. Next, you deduplicate and rerank the merged candidate set using a cross-encoder that scores relevance against the original full question. Finally, you assemble the top chunks into the prompt with clear source attribution so the LLM can synthesize across documents. This iterative retrieval pattern, sometimes called query decomposition or multi-step retrieval, trades higher latency for substantially better recall on complex queries.

Sample Answer

You could use cosine similarity or dot product here. Cosine similarity wins in this case because your embeddings are not L2-normalized, meaning their magnitudes vary. Dot product conflates magnitude with directional similarity, so a longer vector could score higher than a more semantically relevant but shorter one. Cosine similarity divides by the product of norms, $$\text{cos}(\mathbf{a}, \mathbf{b}) = \frac{\mathbf{a} \cdot \mathbf{b}}{\|\mathbf{a}\| \|\mathbf{b}\|}$$, which isolates the angular relationship and gives you a pure measure of semantic alignment. If you later switch to a model that guarantees unit-norm outputs, the two metrics become equivalent and dot product is cheaper to compute.

Sample Answer

Let's reason through this step by step. First, what you are seeing is catastrophic forgetting: the fine-tuning shifted the embedding space to favor support ticket semantics at the expense of the general domain representations the base model had learned. You can confirm this by measuring retrieval quality on a held-out general knowledge eval set before and after fine-tuning and checking whether the embedding distributions have collapsed or shifted using tools like UMAP projections. To fix it, you have a few options: use a lower learning rate with more training steps to gently adapt the model, mix in general-domain contrastive pairs during fine-tuning so the model retains broad coverage, or maintain two separate embedding models and route queries to the appropriate index based on an intent classifier. The routing approach is the most robust in production because it lets you optimize each embedding space independently without compromise.

Sample Answer

First, you want to confirm the hypothesis by pulling the retrieved chunks for failing queries and checking whether the needed information is split across two adjacent chunks. If so, the most likely fix is introducing chunk overlap, typically 10 to 20 percent of the chunk size, so boundary content appears in both neighboring chunks. Next, consider whether your chunk size is too small, because increasing it from say 256 to 512 tokens can reduce the frequency of mid-concept splits. You should also evaluate semantic or structure-aware chunking that respects paragraph and section boundaries rather than splitting at arbitrary token counts. Finally, a complementary approach is retrieving the parent or sibling chunks alongside the top hit at query time, sometimes called contextual or hierarchical retrieval.

Sample Answer

This question is checking whether you understand that metadata is not an afterthought but a first-class lever for retrieval precision. You should extract structural metadata like document title, section headers, page number, and source URL, along with semantic metadata like document type, creation date, and author. This metadata enables filtered or hybrid retrieval where the system can narrow the search space before or during vector similarity search, dramatically reducing false positives. For example, attaching section headers lets you boost chunks from relevant sections, and timestamps let you prefer recent content when freshness matters.

Sample Answer

This question is checking whether you can reason about real engineering tradeoffs, not just recite definitions. IVF partitions the vector space into Voronoi cells using k-means, and at query time you only search a subset of cells ($nprobe$), which gives you a tunable recall-speed knob with relatively low memory overhead since you store raw or quantized vectors without a graph structure. HNSW builds a navigable small-world graph with multiple layers, delivering higher recall at low latency but consuming significantly more memory due to the graph edges (typically 64 to 128 bytes of neighbor pointers per vector). You pick IVF (often with PQ) when memory is constrained or the dataset is very large and you can tolerate slightly lower recall, and you pick HNSW when you need consistently high recall under tight latency budgets and can afford the RAM. In many production RAG systems, a hybrid like IVF-HNSW or IVF-PQ with re-ranking gives you the best of both worlds.

Sample Answer

The standard move is to normalize your vectors to unit length before indexing with inner product, since cosine similarity is equivalent to inner product on $L_2$-normalized vectors: $\cos(\mathbf{a}, \mathbf{b}) = \mathbf{a} \cdot \mathbf{b}$ when $\|\mathbf{a}\| = \|\mathbf{b}\| = 1$. But here, the exception matters: if your embeddings are not normalized, inner product search will favor vectors with larger magnitudes rather than those with the most similar directions, which distorts the neighbor rankings and tanks recall. The fix is either to pre-normalize all vectors before insertion and at query time, or to use an index metric that explicitly handles cosine (which internally normalizes for you). You should also verify that the index was rebuilt after the metric change, since some libraries like FAISS require re-indexing when you switch distance functions.

Sample Answer

The standard move is to add a cross-encoder re-ranker on top of your bi-encoder retrieval because cross-encoders model token-level interactions between query and document, giving much better relevance scores. But here, latency and cost matter because cross-encoders run inference on every (query, document) pair in your candidate set, so if you are re-ranking thousands of candidates or serving sub-100ms latency requirements, the overhead can be prohibitive. You would push back when the initial retriever already achieves strong recall and precision, when latency budgets are tight, or when the candidate pool is too large to re-rank without a first-stage pruning step. The sweet spot is retrieving a broad set (say top 50 to 100 from a fast bi-encoder), then re-ranking to select the final top $k$.

Sample Answer

Get this wrong in production and your legal assistant hallucinates citations or misses critical precedent, which destroys user trust in a high-stakes domain. The right call is to implement an agentic retrieval loop: first, decompose the user query into sub-questions using an LLM, retrieve documents for each sub-question independently, then have the LLM evaluate whether the retrieved context is sufficient to answer the original query. If the sufficiency check fails, you trigger a follow-up retrieval step with a reformulated query, potentially using chain-of-thought reasoning over the partial context to generate more targeted search terms. You should cap the loop at 2 to 3 iterations to control latency and cost, and maintain a running context window that deduplicates retrieved passages across steps.

Sample Answer

Get this wrong in production and your users start trusting fabricated information, which in regulated industries like finance or healthcare can create legal liability. The right call is to build a multi-layer pipeline: first, decompose the generated answer into atomic claims, then use an LLM-as-judge (or a fine-tuned NLI model) to verify each claim against the retrieved context, scoring faithfulness as the fraction of supported claims. You should run this asynchronously on every response, flag outputs below a faithfulness threshold for human review, and log the results to track regression over time. Complement this with a set of curated adversarial test cases where you know the context lacks the answer, verifying the system abstains rather than fabricates. Tools like RAGAS or custom evaluation harnesses built on your own rubrics work well here, but the key is treating this pipeline as a first-class production system with its own monitoring and alerting.

Sample Answer

BLEU and ROUGE sound reasonable but break under the conditions typical of RAG, where correct answers can be paraphrased in many valid ways that share few n-grams with a reference. Exact lexical overlap also does not work because a hallucinated answer could coincidentally match reference tokens while being semantically wrong. That leaves you with semantic evaluation approaches: LLM-as-judge scoring for faithfulness, relevance, and completeness against the retrieved context, combined with embedding-based similarity metrics like BERTScore for softer lexical comparison. You should also maintain a human-labeled golden test set and periodically correlate your automated metrics against human judgments to ensure your evaluation pipeline itself remains calibrated.