AI Engineer Interview Prep

From hundreds of mock interviews, the pattern is clear: candidates who can fine-tune a LoRA adapter but can't wire it into a production API behind a guardrail layer get rejected at the system design round. Across 19 companies we analyzed, LLMs/RAG and ML system design together account for 31% of all interview questions, making applied GenAI fluency the single largest testing surface in the loop.

What AI Engineers Actually Do

AI Engineers build the products that sit between foundation models and end users. You'll find the role at frontier labs like OpenAI and Anthropic, big tech shops like Amazon and Google, fintech companies routing transactions through ML pipelines, and Series B startups shipping their first AI feature with a three-person team. Success after year one means you shipped an AI-powered feature to production, built the eval framework that proves it works, and earned enough trust from product stakeholders that they ask you what's feasible before writing the roadmap.

A Typical Week

The thing that catches most candidates off guard is how much of the job is integration work, not model work. You're debugging flaky CI tests because a mock Bedrock endpoint returns slightly different JSON ordering, refactoring system prompts for an AgentCore routing layer, and reviewing PRs that add Lambda-based guardrails in front of invoke calls. If you imagined yourself heads-down training models five days a week, recalibrate: the AI Engineer's real superpower is stitching together LLM APIs, OpenSearch vector retrieval, DynamoDB metadata stores, and eval harnesses into something that actually works, then explaining the tradeoffs to people who don't know what "chunking strategy" means.

Skills & What's Expected

Three dimensions are rated "expert" across companies: software engineering, machine learning, and modern AI/GenAI. That ML rating comes with teeth (the source descriptions reference PhD-level depth in transformer architectures, reinforcement learning, and neural architecture search), so don't assume interviews stay shallow on theory. What's underrated is the "high" rating on data visualization and business communication. At many companies, you demo working software to engineering leadership, defend your latency and cost-per-invocation numbers live, and write design docs that non-technical stakeholders can follow. If you can't explain why you chose hierarchical chunking over fixed-size chunks in plain English, strong PyTorch skills won't save you.

Levels & Career Growth

Most open roles cluster at Mid and Senior, which is where the market is hungriest right now. The jump from Senior to Staff is the career inflection point that trips people up, because it stops being about how well you build and starts being about cross-team platform decisions: choosing the embedding model and vector store that five product teams depend on, or defining the company's AI safety and cost governance standards. One thing working in your favor: the AI Engineer title is new enough that leveling is inconsistent across companies (some map it to their SWE ladder, others have a separate AI/ML track), and that ambiguity creates real negotiation leverage if you're holding competing offers at different levels.

AI Engineer Compensation

The widget tells you what each level pays, but not why two Senior AI Engineers can be $350K apart. From what candidates report, the gap mostly comes down to employer tier and equity structure. Amazon's 5/15/40/40 RSU schedule, for example, means your Year 1 cash can be 40% below the annualized TC a recruiter quotes, while companies with even four-year vesting deliver a flatter payout. Pre-IPO options add another wrinkle: they're illiquid and pegged to a 409A valuation that may never match the preferred price, so weigh them at a steep discount when comparing offers side by side.

Sign-on bonuses and equity top-ups tend to have the most negotiation room, since base salary bands are usually locked to your level. Showing up with a credible competing offer from even one other company, especially if you've shipped a RAG pipeline or fine-tuning workflow to production, gives recruiters the internal justification to move on those levers. Don't overlook refresh grants either: at large public tech companies, strong performers pick up 20 to 30% of their initial equity grant each year, which quietly compounds your TC well past the number you originally signed.

AI Engineer Interview Process

Across the 18 companies we analyzed, the process converges on about 6 rounds spanning roughly 5 weeks. Where candidates consistently underestimate the difficulty is System Design: they prep for classic distributed-systems prompts but get asked to architect a retrieval-augmented generation pipeline with a vector store like Pinecone, a PyTorch reranker, and an eval harness. From what hiring managers report, this round causes more senior-level rejections than any other.

The final Bar Raiser round trips people up for a different reason. It blends behavioral and system design questions, and the interviewer often probes how you reason through ambiguity rather than whether you nail a textbook answer. Preparing two or three stories where you made a technical judgment call under incomplete information (choosing an embedding model before benchmarks existed, or scoping an LLM feature when product requirements were shifting) will serve you better here than memorizing architecture diagrams.

AI Engineer Interview Questions

LLMs, RAG & Applied AI

This section tests your ability to design and reason about complex AI agents. Expect questions on tool use, context management, and safety principles, which are critical for building capable and reliable systems with models like Claude.

What is RAG (Retrieval-Augmented Generation) and when would you use it over fine-tuning?

EasyFundamentals

Sample Answer

RAG combines a retrieval system (like a vector database) with an LLM: first retrieve relevant documents, then pass them as context to the LLM to generate an answer. Use RAG when: (1) the knowledge base changes frequently, (2) you need citations and traceability, (3) the corpus is too large to fit in the model's context window. Use fine-tuning instead when you need the model to learn a new style, format, or domain-specific reasoning pattern that can't be conveyed through retrieved context alone. RAG is generally cheaper, faster to set up, and easier to update than fine-tuning, which is why it's the default choice for most enterprise knowledge-base applications.

Design a system for an AI agent that acts as a long-term project assistant, needing to recall details from conversations and documents spanning several weeks. How would you manage the agent's context to ensure it has relevant information without exceeding token limits?

AnthropicMediumContext Management & RAG

Sample Answer

I'd implement a hybrid context management system using Retrieval Augmented Generation (RAG). All documents and conversation logs would be chunked, embedded, and stored in a vector database for semantic search. I would also maintain a structured summary of key entities, decisions, and action items that gets updated after each interaction and is always included in the prompt, ensuring the agent has both high-level awareness and the ability to retrieve fine-grained details on demand.

Design a Bedrock AgentCore agent that can issue refunds for Prime orders by calling an internal Refunds API via Lambda, while meeting least-privilege IAM, preventing prompt injection from user text, and ensuring the agent never refunds the wrong order. What is your end-to-end control plan (auth, tool schema, validation, logging, and human-in-the-loop) and where do you put each control?

AmazonHardAgent Tool Use Safety and Governance

Sample Answer

Unlock the full solution

View detailed answers and practice in our question bank

View Solution

Practice more LLMs, RAG & Applied AI questions

Ml System Design

This section checks whether you can take an LLM from dataset to production and keep it stable under real traffic. You will be judged on data quality, training and serving architecture, and reliability tradeoffs like latency, cost, and safety.

Design a Bedrock Knowledge Bases based RAG assistant for the company Seller Support that answers policy questions from 5 million PDFs in S3 with a p95 latency under 2 seconds. Specify chunking, embedding refresh strategy, OpenSearch vector index design, and how you prevent outdated answers after daily policy updates.

AmazonMediumRAG Architecture and Indexing

Sample Answer

Most candidates default to embedding everything nightly and using top-$k$ cosine search, but that fails here because daily policy changes create stale vectors and top-$k$ alone pulls near-duplicates that waste context. You need an incremental ingestion path keyed by document version, with delete and upsert semantics in the vector index and a freshness filter (policy effective date) applied at retrieval time. Use chunking tuned to policy structure (section headers, bullet lists), store chunk metadata (doc_id, version, effective_date, locale), and add an MMR or diversification step to avoid redundant chunks. For correctness, gate answers with citations, add a fallback to keyword search for exact policy terms, and block generation when retrieved context is below a similarity threshold.

Design an agentic RAG workflow using Bedrock AgentCore that can call internal tools (order status API, returns policy lookup) for the company customer service, and must meet compliance requirements (PII redaction, audit logs) while limiting hallucinations. Specify orchestration, guardrails, tool schemas, and observability signals that would trigger rollback.

AmazonHardAgentic RAG and Safety

Sample Answer

Unlock the full solution

View detailed answers and practice in our question bank

View Solution

Practice more Ml System Design questions

Machine Learning & Modeling

Your ability to reason about learning objectives, generalization, and optimization trade-offs is a primary signal for research credibility. You’ll be pushed past definitions into “why it works/when it fails” arguments and ablations you’d run.

What is the bias-variance tradeoff?

EasyFundamentals

Sample Answer

Bias is error from oversimplifying the model (underfitting) — a linear model trying to capture a nonlinear relationship. Variance is error from the model being too sensitive to training data (overfitting) — a deep decision tree that memorizes noise. The tradeoff: as you increase model complexity, bias decreases but variance increases. The goal is to find the sweet spot where total error (bias squared + variance + irreducible noise) is minimized. Regularization (L1, L2, dropout), cross-validation, and ensemble methods (bagging reduces variance, boosting reduces bias) are practical tools for managing this tradeoff.

Your SageMaker endpoint for a fraud model shows AUC flat week over week, but the fraud catch rate at a fixed alert volume dropped after a policy change. What statistical checks do you run to diagnose what changed, and what metric or thresholding change do you make?

AmazonMediumMonitoring, Drift, and Thresholding

Sample Answer

Get this wrong in production and you either miss fraud (loss) or flood operations with false alerts (cost), with AUC hiding the damage. The right call is to check label shift and score calibration drift (PR curve, precision at $k$, calibration curve, PSI on features and on scores), then reset thresholds against the new base rate and optimize for an ops-constrained metric like precision at fixed volume or cost-weighted expected value.

You evaluate a new RAG retriever for a Bedrock chatbot and see higher average cosine similarity of retrieved chunks, but worse human rated answer quality. What is the most likely failure mode, and how do you redesign the evaluation to select the better retriever?

AmazonHardError Analysis and Retrieval Evaluation

Sample Answer

Unlock the full solution

View detailed answers and practice in our question bank

View Solution

Practice more Machine Learning & Modeling questions

Deep Learning

Explain why LayerNorm is typically preferred over BatchNorm in transformer blocks, and what breaks when you crank microbatch size down to 1 or use gradient accumulation.

MistralMediumNormalization in Deep Networks

Sample Answer

BatchNorm depends on accurate batch statistics, so tiny batches make its mean and variance noisy, which destabilizes training and creates train eval mismatch. Gradient accumulation does not fix BN stats, it only changes the effective batch for gradients, not for normalization. LayerNorm normalizes per token (or per sample) across features, so it is stable with batch size 1 and works cleanly with accumulation. That is why transformer training at scale almost always uses LayerNorm or RMSNorm.

In the company Photos, you build a multimodal retrieval model (text query to image search) using a CLIP-style dual encoder and observe high Recall@10 offline but users report irrelevant results for rare entities and long queries. Walk through, step by step, how you would diagnose whether the failure is in embedding geometry, fusion/tokenization, or hard negative mining, and what architecture-level changes you would try first.

AppleHardMultimodal Retrieval and Embedding Geometry

Sample Answer

Unlock the full solution

View detailed answers and practice in our question bank

View Solution

Practice more Deep Learning questions

Coding & Algorithms

Your ability to write correct, efficient code under time pressure is still a core gate, even for an AI-focused role. The bar is clean reasoning about complexity, edge cases, and implementation details—not clever tricks.

For an Bedrock Knowledge Base, you ingest $n$ documents each with an embedding vector; for each doc you also store up to 50 near-duplicates detected by cosine similarity, forming an undirected graph. Implement `count_components(n, edges)` that returns the number of connected components so you can batch dedup jobs per component, where `edges` is a list of pairs $(u, v)$ with $0 \le u, v < n$.

AmazonMediumGraph Traversal, Union Find

Sample Answer

Unlock the full solution

View detailed answers and practice in our question bank

View Solution

An AI agent can use a set of tools, where each tool has a cost and a list of dependency tools that must be executed first. Given a target tool, find the minimum cost to execute it, including the costs of all its direct and indirect dependencies.

AnthropicHardGraph Traversal

Sample Answer

This problem can be modeled as finding the cost of a path in a Directed Acyclic Graph (DAG), where tools are nodes and dependencies are edges. We can solve this efficiently using recursion with memoization (a form of dynamic programming) to avoid recomputing costs for the same tool. The base case is a tool with no dependencies, and the recursive step sums the tool's own cost with the minimum costs of its dependencies.

PythonCopy

1from typing import Dict, List, Optional
2
3def find_minimum_cost(target_tool: str, tools: Dict[str, Dict], memo: Dict[str, int]) -> Optional[int]:
4    """Calculates the minimum cost to run a target tool including dependencies.
5
6    Args:
7        target_tool (str): The name of the tool to execute.
8        tools (Dict): A dictionary defining all tools, their costs, and dependencies.
9        memo (Dict): A memoization cache to store computed costs.
10
11    Returns:
12        Optional[int]: The minimum cost, or None if a dependency is missing.
13    """
14    # Return cached result if already computed
15    if target_tool in memo:
16        return memo[target_tool]
17
18    # Check if the tool exists
19    if target_tool not in tools:
20        return None # Or raise an error for an invalid tool
21
22    tool_info = tools[target_tool]
23    current_cost = tool_info['cost']
24    
25    # Recursively calculate cost of dependencies
26    for dep in tool_info['dependencies']:
27        dep_cost = find_minimum_cost(dep, tools, memo)
28        if dep_cost is None:
29            # A dependency is missing, so this path is impossible
30            return None
31        current_cost += dep_cost
32
33    # Cache the result before returning
34    memo[target_tool] = current_cost
35    return current_cost
36
37# Example Usage:
38tools_definition = {
39    'get_weather': {'cost': 10, 'dependencies': []},
40    'get_location': {'cost': 5, 'dependencies': []},
41    'plan_trip': {'cost': 20, 'dependencies': ['get_weather', 'get_location']},
42    'book_flight': {'cost': 50, 'dependencies': ['plan_trip']},
43    'send_confirmation': {'cost': 2, 'dependencies': ['book_flight']}
44}
45
46target = 'send_confirmation'
47memo_cache = {}
48total_cost = find_minimum_cost(target, tools_definition, memo_cache)
49
50if total_cost is not None:
51    print(f"Minimum cost to run '{target}': {total_cost}")
52else:
53    print(f"Could not compute cost for '{target}' due to missing dependencies.")
54
55# Expected output: 87 (2 + 50 + 20 + 10 + 5)
56

Practice more Coding & Algorithms questions

Engineering

Your AI service calls the company Data Cloud query APIs to fetch features for real-time lead scoring, and you are hitting rate limits and occasional 5xx from upstream. How do you design retries, backoff, and circuit breaking so you protect the company and still meet a p95 latency SLO for scoring?

SalesforceMediumResilience, Rate Limits, and Backpressure

Sample Answer

Start with what the interviewer is really testing: "This question is checking whether you can build a dependency-safe integration that fails predictably under CRM-scale load." You cap retries, use exponential backoff with jitter, and you only retry on known transient classes (timeouts, 429, selected 5xx), otherwise fail fast. You add a circuit breaker per upstream endpoint and per tenant to prevent retry storms, and you shed load by returning a degraded score with a freshness flag when Data Cloud is unavailable. You instrument p95 and error budgets, then tune concurrency and retry budgets so worst-case retries cannot blow your latency SLO.

You are serving a 7B LLM on an L4 with TensorRT-LLM, and p99 latency spikes when you enable continuous batching even though throughput improves. What GPU-side mechanisms cause this (KV cache growth, attention kernel shape changes, workspace pressure, stream contention), and what concrete changes would you make to recover p99 without giving back most throughput?

NvidiaHardInference Performance and Scheduling

Sample Answer

Unlock the full solution

View detailed answers and practice in our question bank

View Solution

Practice more Engineering questions

Cloud Infrastructure

In practice, you’ll need to explain how an LLM service stays up when traffic spikes, dependencies fail, or models change. You’ll be evaluated on deployment patterns, observability, rollout strategies, and securing/isolating enterprise workloads.

You need to run an ingestion pipeline that chunks PDFs from S3, creates embeddings, and indexes them into OpenSearch, triggered when new objects arrive. Would you implement this with Step Functions plus Lambda, or EventBridge Pipes directly to a compute target, and why?

AmazonMediumServerless Orchestration

Sample Answer

You could do Step Functions plus Lambda, or EventBridge Pipes directly to a target like Lambda or ECS. Step Functions wins here because ingestion needs explicit state, retries per step, error handling branches, and idempotency checkpoints, especially when chunking and embedding can partially fail. Pipes wins when it is a straight-through transform and deliver path with minimal orchestration, low latency, and simple retry semantics. For production RAG ingestion, you usually need the visibility and control of a state machine, not just wiring.

Your Bedrock-based chat assistant runs on Lambda behind API Gateway, p95 latency is spiking, and CloudWatch shows intermittent throttles and timeouts during traffic bursts. How do you harden the AWS deployment for scaling and operational readiness without changing the model or prompts?

AmazonHardScalability and Operations

Sample Answer

Unlock the full solution

View detailed answers and practice in our question bank

View Solution

Practice more Cloud Infrastructure questions

Ml Operations

The bar here isn’t whether you know MLOps terms, it’s whether you can operationalize ML with reproducibility, CI/CD, and observability. You’ll be pressed on how you handle data/model drift, versioning, retraining triggers, and incident response.

You need reproducible model promotion across dev, staging, and prod for a SageMaker endpoint that serves an embedding model used by OpenSearch vector search. How do you version data, code, and model artifacts, and what CI/CD gates do you add so a bad embedding change cannot silently degrade recall?

AmazonMediumModel Versioning and CI/CD Gates

Sample Answer

The standard move is to version every artifact, dataset snapshot identifiers, training code commit SHA, container image digest, and model package version in a registry, then promote only immutable references through environments. But here, retrieval quality matters because embedding drift can look like a backend issue while it actually breaks nearest-neighbor geometry, so you gate on offline retrieval metrics like Recall@$k$, nDCG@$k$, and an embedding distribution check against a baseline. You also add contract tests for vector dimension, normalization, and latency, plus a shadow or canary evaluation on live queries before full rollout. If the CI/CD pipeline cannot recreate the exact model from metadata, you do not have real versioning.

A Bedrock AgentCore assistant starts hallucinating policy answers after a Knowledge Base reindex and the support team opens a Sev2. What do you change in your monitoring and incident playbook to detect this within 5 minutes and isolate whether the failure is retrieval, prompt, or model behavior?

AmazonHardLLM Ops Incident Response and Drift Detection

Sample Answer

Unlock the full solution

View detailed answers and practice in our question bank

View Solution

Practice more Ml Operations questions

The distribution skews heavily toward applied GenAI and architectural thinking, which means your prep should too. When a system design question asks you to build a Bedrock-based RAG assistant over millions of PDFs, you can't separate the "design" from the "LLM knowledge." You need to justify your chunking strategy, defend why you chose a cross-encoder reranker over a bi-encoder, and explain your RAGAS eval metrics, all while sketching the serving architecture. The single biggest prep mistake is defaulting to algorithm grinding out of comfort, when the questions that actually separate candidates require you to, say, explain why your OpenSearch vector index needs a different refresh cadence than your SageMaker embedding endpoint's autoscaling policy.

Practice questions across all eight topic areas at datainterview.com/questions.

How to Prepare

Stop prepping for this like a standard software engineering loop. LLMs/RAG and ML system design questions show up more than any other category in AI Engineer interviews, so front-load your study plan accordingly. Your opening two weeks should focus on transformer internals (attention mechanisms, positional encodings, KV caching), retrieval architectures (chunking strategies, hybrid search, cross-encoder reranking), and embedding model selection (BGE, E5, OpenAI's text-embedding-3-large). Practice whiteboarding system design prompts out loud: "Design a RAG pipeline that ingests 50K PDFs from S3, indexes them in Pinecone, and serves answers with sub-2-second latency" is the kind of scenario you should be able to riff on for 45 minutes.

Shift your final two weeks toward coding, deep learning theory, and deployment. Solve two medium-difficulty Python problems per day (trees, graphs, dynamic programming) while reviewing backprop, batch norm, and CNN vs. RNN vs. transformer tradeoffs in parallel. On the infrastructure side, practice containerizing a PyTorch model with Docker, writing a GitHub Actions workflow for model retraining, and deploying to a SageMaker endpoint.

Build one small end-to-end project before your first interview: take a PDF corpus, pipe it through LangChain or LlamaIndex with a FAISS or Chroma vector store, deploy it behind a FastAPI endpoint, and add evals with RAGAS. Write four STAR-format stories that each highlight a different signal (technical judgment, cross-team influence, shipping under ambiguity, handling a model failure in production) and run at least two timed mock interviews covering both system design and behavioral rounds.

Try a Real Interview Question

1from typing import Dict, List
2
3
4def top_k_cosine_sparse(query: Dict[int, float], candidates: List[Dict[int, float]], k: int) -> List[int]:
5    """Return indices of the top-k candidates by cosine similarity to a sparse query vector.
6
7    Args:
8        query: Sparse vector as {dimension_index: value}.
9        candidates: List of sparse vectors in the same format.
10        k: Number of indices to return.
11
12    Returns:
13        List of candidate indices sorted by decreasing cosine similarity, tie-breaking by smaller index.
14    """
15    pass
16

1from typing import Dict, List, Tuple
2import heapq
3import math
4
5
6def _sparse_norm(v: Dict[int, float]) -> float:
7    s = 0.0
8    for x in v.values():
9        s += x * x
10    return math.sqrt(s)
11
12
13def _sparse_dot(a: Dict[int, float], b: Dict[int, float]) -> float:
14    if len(a) > len(b):
15        a, b = b, a
16    s = 0.0
17    for i, va in a.items():
18        vb = b.get(i)
19        if vb is not None:
20            s += va * vb
21    return s
22
23
24def top_k_cosine_sparse(query: Dict[int, float], candidates: List[Dict[int, float]], k: int) -> List[int]:
25    """Return indices of the top-k candidates by cosine similarity to a sparse query vector.
26
27    Args:
28        query: Sparse vector as {dimension_index: value}.
29        candidates: List of sparse vectors in the same format.
30        k: Number of indices to return.
31
32    Returns:
33        List of candidate indices sorted by decreasing cosine similarity, tie-breaking by smaller index.
34    """
35    n = len(candidates)
36    if k <= 0 or n == 0:
37        return []
38
39    qn = _sparse_norm(query)
40    if qn == 0.0:
41        return list(range(min(k, n)))
42
43    heap: List[Tuple[float, int]] = []
44    for idx, cand in enumerate(candidates):
45        cn = _sparse_norm(cand)
46        if cn == 0.0:
47            sim = 0.0
48        else:
49            sim = _sparse_dot(query, cand) / (qn * cn)
50
51        key = (sim, -idx)
52        if len(heap) < k:
53            heapq.heappush(heap, key)
54        else:
55            if key > heap[0]:
56                heapq.heapreplace(heap, key)
57
58    best = sorted(heap, key=lambda t: (-t[0], -t[1]))
59    return [-t[1] for t in best]
60

AI Engineer coding rounds rarely ask you to just invert a binary tree. You're more likely to manipulate embedding vectors, parse structured LLM outputs, or implement a scoring function that mirrors a real retrieval pipeline. Practice more problems tuned to this format at datainterview.com/coding.

Test Your Readiness

Use your results to pinpoint weak spots before locking in a study plan. For targeted practice across every topic area, explore datainterview.com/questions.