PyTorch interviews rarely ask you to define a tensor. They hand you a training loop that runs but learns nothing, a job that runs out of memory at step 2,000, or a DDP launch that hangs, and ask you to find the cause. Machine learning engineers, research engineers and applied scientists get these questions because the bugs live in the details of how PyTorch stores data and records gradients.
Candidates usually slip on the mechanics under the API. Which operations return views? What does model.eval() leave unchanged? Why does torch.load now refuse some checkpoints, and where does GPU memory actually go? A strong answer names the fix and then explains what PyTorch is doing underneath.
The 30 questions below are grouped into five areas: tensors, autograd, training loops, performance, and distributed training with deployment. Each solution gives the answer, the reason and the trap, with short code checked against current PyTorch 2.x. A preparation plan, a quiz and an FAQ follow the questions.
Intermediate30 questions
PyTorch Interview Questions
Practice questions on how PyTorch stores tensors, computes gradients and trains at scale, with answers checked against current PyTorch 2.x.
Many PyTorch bugs that survive code review are shape and memory bugs. Interviewers use tensor questions to find out whether you know what a tensor is underneath the API: a storage buffer plus a shape, one stride per dimension and an offset.
The idea that matters most is that many operations return views sharing storage with their input. Once you can predict which calls alias and which ones copy, errors like view size is not compatible and silent in-place corruption stop being mysterious.
Tensors, Strides and Views
You transpose a (3, 4) tensor and call y.view(-1) to flatten it before a linear layer. PyTorch raises view size is not compatible with input tensor's size and stride. What happened, and how do you fix it?
EasyTensors, Strides and Views
Sample Answer
torch.arange(12).reshape(3, 4) has strides (4, 1): one step along a row moves 1 element in memory, one step down a column moves 4. .t() copies nothing. It swaps the strides to (1, 4), so the tensor is no longer contiguous in row-major order. view can only reinterpret shape and strides over the same memory, and no single stride walks the transposed elements in order, so it fails. Use y.reshape(-1), which returns a view when the strides allow it and copies otherwise, or y.contiguous().view(-1) to make the copy explicit. The trap is assuming reshape always aliases. After a copy, in-place writes to the result no longer reach the original.
A teammate saves the first 10 rows of a large embedding table with torch.save(emb[:10], 'head.pt'). The file is as big as the whole table. Why, and what should they save instead?
MediumTensors, Strides and Views
Sample Answer
Slicing returns a view: the same storage with a new shape, strides and storage_offset. torch.save preserves view relationships by writing the underlying storage, so a 10-row view of a 1000 x 1000 float32 table writes the full 4 MB buffer. Call .clone() first. It allocates compact storage holding only the slice, and the file drops to about 40 KB. The same aliasing causes a quieter memory leak in training code: keeping a small slice of a large activation alive keeps the entire activation in memory. Clone anything you plan to hold onto long term so the parent buffer can be freed.
Python
1import io, torch
2big = torch.randn(1000,1000)3for t in(big[:10], big[:10].clone()):4 buf = io.BytesIO()5 torch.save(t, buf)6print(buf.tell())# about 4,001,000 bytes, then about 41,000
Your regression model trains, but the loss plateaus at a suspiciously high value. The model outputs shape (N, 1) and the targets have shape (N,). What is going on?
EasyTensors, Strides and Views
Sample Answer
Broadcasting. Subtracting an (N,) tensor from an (N, 1) tensor aligns trailing dimensions and produces an (N, N) matrix, so the loss averages the error of every prediction against every target. The model is pulled toward the mean of all targets and the loss stalls. nn.MSELoss does emit a UserWarning that the target size differs from the input size, but a warning in a long training log is easy to miss. Fix it with pred.squeeze(-1) or target.unsqueeze(-1), and put assert pred.shape == target.shape before the loss. Interviewers like this bug because nothing crashes.
Python
1import torch
2import torch.nn.functional as F
3pred, target = torch.randn(4,1), torch.randn(4)4print((pred - target).shape)# torch.Size([4, 4])5loss = F.mse_loss(pred.squeeze(-1), target)
You need to add a per-channel bias of shape (C,) to a batch of shape (B, C, H, W). Compare expand with repeat, and explain why calling .add_(1) on the expanded bias fails.
MediumTensors, Strides and Views
Sample Answer
expand returns a view with stride 0 along the broadcast dimensions, so it allocates nothing: every position reads the same element. repeat materializes a full B*C*H*W copy. For the addition you need neither, because x + bias.view(1, C, 1, 1) broadcasts on its own. The in-place call fails because many logical positions of the expanded tensor map to one memory location, and PyTorch refuses with unsupported operation: more than one element of the written-to tensor refers to a single memory location. Clone first if you really need a writable expanded tensor. In short, broadcasting and expand cost no memory, while repeat is a real allocation.
Python
1import torch
2B, C, H, W =8,3,32,323x, bias = torch.randn(B, C, H, W), torch.randn(C)4e = bias.view(1, C,1,1).expand(B, C, H, W)5print(e.stride())# (0, 1, 0, 0): no new memory6y = x + bias.view(1, C,1,1)# broadcasting does the same job
An image pipeline loads pixels as uint8 and computes (img - 128) / 128 to center them. The model trains, but poorly. What went wrong?
MediumTensors, Strides and Views
Sample Answer
Type promotion with Python scalars. A uint8 tensor combined with a Python int stays uint8, so img - 128 wraps modulo 256: a pixel of 10 becomes 138 instead of -118. The division then promotes to float32, which hides the bug because the output looks like ordinary floats. Convert first with img.float(), then subtract and divide. The same rule matters in mixed precision. A float16 tensor plus a Python float, or plus a 0-dim float32 tensor, stays float16, while a float16 tensor plus a float32 tensor that has dimensions promotes to float32. Print result.dtype whenever precision matters.
Given log-probabilities logp of shape (N, K) and integer labels y of shape (N,), compute the negative log-likelihood without calling a loss function.
MediumTensors, Strides and Views
Sample Answer
Pick each row's entry at its label. logp.gather(1, y.unsqueeze(1)) indexes dimension 1 with an (N, 1) index tensor and returns (N, 1), so squeeze it back. Advanced indexing, logp[torch.arange(N), y], gives the same values and reads more clearly. Negate and take the mean, which matches F.nll_loss(logp, y) with its default reduction='mean'. The traps are shape and dtype: gather needs an int64 index with the same number of dimensions as the input, so passing y without unsqueeze raises Index tensor must have the same number of dimensions as input tensor.
Python
1import torch
2import torch.nn.functional as F
3logp = torch.randn(4,5).log_softmax(dim=1)4y = torch.tensor([0,3,1,4])5nll =-logp.gather(1, y.unsqueeze(1)).squeeze(1).mean()6same =-logp[torch.arange(4), y].mean()7print(torch.allclose(nll, F.nll_loss(logp, y)), torch.allclose(nll, same))
Autograd questions separate people who call loss.backward() from people who can debug it. Expect errors raised in the backward pass, gradients that come out as None or zero, and prompts that ask you to write your own backward.
The core idea: during the forward pass PyTorch records a graph and saves the tensors each backward formula needs. Most autograd errors are about that saved state. It was freed, it was modified in place, or it was never recorded.
Autograd and Gradients
You compute two losses from one forward pass and call loss1.backward() and then loss2.backward(). The second call raises Trying to backward through the graph a second time. Why, and what is the right fix?
EasyAutograd and Gradients
Sample Answer
After a backward pass, autograd frees the intermediate tensors it saved for computing gradients, to reclaim memory. The second call reaches the shared part of the graph and finds those buffers gone. The usual fix is (loss1 + loss2).backward(): one traversal, the same gradients. loss1.backward(retain_graph=True) also works, but it keeps the saved tensors alive and costs memory. Also remember that gradients accumulate. Two backward calls add into .grad rather than overwrite it, which is why training loops zero gradients every step and why gradient accumulation works at all.
Python
1import torch
2x = torch.randn(3, requires_grad=True)3h = x.exp()4loss1, loss2 = h.sum(),(2* h).sum()5(loss1 + loss2).backward()# one pass through the shared graph6print(x.grad)# equals 3 * exp(x)
During validation, GPU memory spikes and the job runs out of memory even though you called model.eval(). One colleague says to add torch.no_grad(), another says torch.inference_mode(). What does each of the three do?
MediumAutograd and Gradients
Sample Answer
model.eval() only sets training=False on every module, which changes Dropout and BatchNorm. It does not stop autograd from recording the graph, so activations are still saved and memory grows. torch.no_grad() turns off graph recording, so outputs have requires_grad=False. torch.inference_mode() goes further and also skips view tracking and version counter bumps, which makes it a bit faster. The catch is that tensors created inside it are inference tensors: using one later in a computation autograd must record raises Inference tensors cannot be saved for backward. For evaluation, use model.eval() together with one of the two context managers.
Python
1import torch
2lin = torch.nn.Linear(3,3)3lin.eval()4print(lin(torch.randn(2,3)).requires_grad)# True: eval() does not stop autograd5with torch.inference_mode():6 out = lin(torch.randn(2,3))7print(out.requires_grad, out.is_inference())# False True
A codebase uses y.data in several places to modify activations "without affecting gradients". You propose replacing it with y.detach(). Explain the difference and why .data is dangerous.
HardAutograd and Gradients
Sample Answer
Both return a tensor that shares storage with y and has no graph history. The difference is the version counter. detach() shares it, so if you modify the detached tensor in place and y was saved for backward, autograd notices and raises one of the variables needed for gradient computation has been modified by an inplace operation. .data hides the change from autograd, and backward runs on the corrupted values. In the snippet, sigmoid's backward reuses its saved output, so zeroing that output through .data silently produces zero gradients. Use detach(), and detach().clone() when you need an independent copy.
Python
1import torch
2x = torch.randn(3, requires_grad=True)3y = x.sigmoid()4y.data.zero_()# autograd is not told5y.sum().backward()6print(x.grad)# tensor([0., 0., 0.]): silently wrong7# y.detach().zero_() instead makes backward() raise a RuntimeError
Write a manual SGD update for a tensor w created with requires_grad=True. Why does a bare w -= lr * w.grad raise an error?
EasyAutograd and Gradients
Sample Answer
w is a leaf tensor that requires grad. Modifying it in place while autograd is recording would put the update itself into the graph and overwrite a value autograd may need, so PyTorch raises a leaf Variable that requires grad is being used in an in-place operation. Wrap the update in torch.no_grad() so it is not recorded, then clear the gradient so the next backward does not add to the old one. Writing w = w - lr * w.grad avoids the error but is worse: it rebinds w to a new non-leaf tensor, and anything that held the original parameter, such as an optimizer, no longer sees the update. This no-grad, in-place update is what torch.optim optimizers do internally.
Implement a rounding op for quantization-aware training whose gradient passes straight through, using a custom torch.autograd.Function.
HardAutograd and Gradients
Sample Answer
torch.round has a zero gradient almost everywhere, so nothing upstream learns through it. A straight-through estimator rounds in the forward pass and acts as the identity in backward. Subclass torch.autograd.Function, define forward and backward as static methods, and call it with .apply, never by instantiating the class. backward returns one gradient per input of forward, with None for non-tensor arguments. If backward needs values from forward, store them with ctx.save_for_backward. For ops that should have exact gradients, check them with torch.autograd.gradcheck on float64 inputs; an STE fails that check by design.
For a probing experiment you need the activations of an intermediate layer of a pretrained model, and you cannot edit the model's code. How do you capture them, and what memory trap do you avoid?
MediumAutograd and Gradients
Sample Answer
Register a forward hook on the submodule. module.register_forward_hook(fn) calls fn(module, inputs, output) after each forward of that module and returns a handle. Store output.detach(). The raw output keeps a reference to the autograd graph, and over many batches that keeps every saved activation alive. Call handle.remove() when you are done, because hooks persist and stack up if you register them twice. For gradients, tensor.register_hook lets you inspect or modify a gradient as it flows back, and module.register_full_backward_hook gives gradients with respect to a module's inputs and outputs.
Python
1import torch
2import torch.nn as nn
3net = nn.Sequential(nn.Linear(4,8), nn.ReLU(), nn.Linear(8,2))4acts ={}5defsave(module, inputs, output):6 acts["relu"]= output.detach()7handle = net[1].register_forward_hook(save)8net(torch.randn(5,4))9handle.remove()10print(acts["relu"].shape)# torch.Size([5, 8])
Every PyTorch role expects you to write a training loop from memory and to spot what is wrong with someone else's. The bugs interviewers plant are small: a missing zero_grad, a model left in training mode, a tensor that never reached the GPU.
What matters most is the state a module carries besides its weights, such as the training flag, buffers like BatchNorm's running statistics, and which attributes are registered at all. Most loop bugs come from state you forgot was there.
nn.Module and Training Loops
Validation accuracy changes every time you evaluate the same checkpoint on the same data. What is the most likely cause, and what does model.eval() change?
Easynn.Module and Training Loops
Sample Answer
The model is still in training mode. There, nn.Dropout zeroes activations at random (p=0.5 by default) and scales the rest by 1/(1-p), so predictions vary run to run. nn.BatchNorm normalizes with the current batch's statistics and updates its running mean and variance with momentum=0.1, so evaluating also changes the model. model.eval() sets training=False recursively: Dropout becomes the identity and BatchNorm uses its stored running statistics. It does not disable gradient tracking, so pair it with torch.no_grad() or torch.inference_mode(), and call model.train() before the next training epoch.
Write a training step that accumulates gradients over 4 micro-batches to simulate a batch 4 times larger. Which mistakes are common?
Mediumnn.Module and Training Loops
Sample Answer
Call backward on every micro-batch and step once every 4. Divide each loss by the accumulation count so the summed gradient equals the gradient of the mean loss over the large batch; without it the effective learning rate is 4 times higher. Zero gradients after each optimizer step, not before every backward. zero_grad() defaults to set_to_none=True, which frees the gradient tensors. Clip after the last backward and before step(). Two subtler points. BatchNorm still sees only the micro-batch, so its statistics differ from a true large batch. And when the number of batches is not a multiple of 4, the final partial window still needs a step.
Python
1import torch
2import torch.nn as nn
3model, loss_fn = nn.Linear(10,1), nn.MSELoss()4opt = torch.optim.SGD(model.parameters(), lr=0.01)5loader =[(torch.randn(8,10), torch.randn(8,1))for _ inrange(10)]6accum =47for i,(x, y)inenumerate(loader):8 loss = loss_fn(model(x), y)/ accum
9 loss.backward()10if(i +1)% accum ==0or i +1==len(loader):11 nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)12 opt.step()13 opt.zero_grad()
After model.to('cuda'), the forward pass fails with Expected all tensors to be on the same device. In __init__ the model stores a mask as self.mask = torch.ones(10). What is wrong, and how should it be stored?
Mediumnn.Module and Training Loops
Sample Answer
Module.to() moves parameters and buffers. A plain tensor attribute is neither, so self.mask stays on the CPU. Register it with self.register_buffer('mask', torch.ones(10)). Buffers move with .to(), get cast by .half() if they are floating point, and appear in state_dict(); pass persistent=False to keep one out of checkpoints. The same registration rule explains a second classic bug: layers kept in a plain Python list are invisible to parameters(), so the optimizer never updates them. Use nn.ModuleList or nn.ModuleDict. Also note that Module.to() works in place, while Tensor.to() returns a new tensor you must assign.
Python
1import torch
2import torch.nn as nn
3classBad(nn.Module):4def__init__(self):5super().__init__()6 self.layers =[nn.Linear(2,2)for _ inrange(3)]# not registered7 self.mask = torch.ones(2)# not moved by .to()8 self.register_buffer("ok_mask", torch.ones(2))9m = Bad()10print(len(list(m.parameters())),list(m.state_dict()))# 0 ['ok_mask']
You fine-tune a pretrained CNN with 2 images per GPU and accuracy falls well below the frozen-feature baseline. One run also crashed in the classifier head with Expected more than 1 value per channel when training. Diagnose both.
Hardnn.Module and Training Loops
Sample Answer
BatchNorm in training mode normalizes with the current batch's statistics. With 2 images those estimates are very noisy, and the running averages, updated with momentum=0.1, drift away from the pretrained values. The crash comes from a BatchNorm1d after pooling: a last batch with one sample has a single value per channel, so no variance exists. Fixes: keep BatchNorm layers in eval mode during fine-tuning (and optionally freeze their affine parameters), set drop_last=True on the DataLoader, convert to nn.SyncBatchNorm under DDP so statistics span all GPUs, or use GroupNorm or LayerNorm, which do not depend on batch size.
Python
1import torch.nn as nn
2BN =(nn.BatchNorm1d, nn.BatchNorm2d, nn.BatchNorm3d)3deftrain_with_frozen_bn(model):4 model.train()5for m in model.modules():6ifisinstance(m, BN):7 m.eval()# use the pretrained running statistics8 m.requires_grad_(False)# optional: freeze weight and bias too
A classifier ends in nn.Softmax and is trained with nn.CrossEntropyLoss. Training is slow and the loss plateaus high. What is wrong, and what inputs and targets does CrossEntropyLoss expect?
Easynn.Module and Training Loops
Sample Answer
CrossEntropyLoss expects raw logits. It applies log_softmax internally, fused with the negative log-likelihood for numerical stability. Feeding it probabilities applies softmax twice, which squashes outputs toward uniform and shrinks the gradients. Remove the softmax layer and apply it only when you need probabilities at inference. Targets are either class indices of shape (N,) with dtype int64 (a float tensor of that shape raises a dtype error) or class probabilities of shape (N, C) for soft labels. The label_smoothing argument and ignore_index, which defaults to -100 and is handy for padding tokens, cover the common variants.
You resume training from a checkpoint and the loss jumps back up for a few hundred steps. Separately, a teammate's checkpoint fails to load with an UnpicklingError. Explain both and show how you save and load a checkpoint.
Mediumnn.Module and Training Loops
Sample Answer
Resuming needs more than weights. AdamW keeps per-parameter moment estimates and the scheduler keeps its step count, so restoring only model.state_dict() restarts both and causes the jump. Save the optimizer and scheduler state and the epoch too. The load error comes from torch.load, whose weights_only default became True in PyTorch 2.6. It only unpickles tensors, primitive containers and allow-listed types, so a checkpoint holding a custom Python object fails. Store plain dicts, or allow-list trusted classes with torch.serialization.add_safe_globals; use weights_only=False only for files you trust. Under DDP, save model.module.state_dict() to avoid a module. prefix on every key.
Senior ML engineering interviews spend real time on why a job is slow or out of memory. Interviewers want a method: measure, name the bottleneck, then choose the fix that targets it.
Keep the budgets separate. A training step spends time in the input pipeline and in GPU compute, and its memory splits into weights, gradients, optimizer state and activations. Each fix in this section moves one of those numbers.
Performance, Memory and torch.compile
GPU utilization hovers around 30% while training an image model, and the GPU idles between bursts. How do you confirm the input pipeline is the bottleneck, and which DataLoader settings do you change?
MediumPerformance, Memory and torch.compile
Sample Answer
Measure first: time an epoch that only iterates the DataLoader, or profile a few steps with torch.profiler and look for gaps where the GPU waits for the next batch. If loading is the bottleneck, raise num_workers from its default of 0, which loads on the main process, toward the CPU cores available per GPU. Set pin_memory=True and call .to(device, non_blocking=True) so host-to-device copies run asynchronously. persistent_workers=True avoids restarting workers every epoch, and prefetch_factor (2 batches per worker by default once workers exist) sets read-ahead. If decoding or augmentation is still too slow, move it to the GPU. PyTorch seeds torch, random and NumPy differently in each worker, but other libraries need a worker_init_fn.
Convert an FP32 training loop to mixed precision. When do you need a GradScaler, and why is model.half() the wrong approach?
MediumPerformance, Memory and torch.compile
Sample Answer
Wrap the forward pass and loss in torch.autocast. Autocast runs matmuls and convolutions in low precision, keeps precision-sensitive ops such as softmax, norms and most losses in float32, and leaves the parameters in float32. With torch.float16, small gradients underflow to zero, so torch.amp.GradScaler('cuda') multiplies the loss before backward, unscales before the step, and skips steps whose gradients contain inf or NaN. With torch.bfloat16, which shares float32's exponent range, you can drop the scaler. model.half() converts the weights themselves, so updates smaller than float16's resolution vanish. Call scaler.unscale_(opt) before clipping. torch.cuda.amp.GradScaler is deprecated in favor of torch.amp.GradScaler.
Python
1scaler = torch.amp.GradScaler("cuda")2for x, y in loader:3 opt.zero_grad()4with torch.autocast(device_type="cuda", dtype=torch.float16):5 loss = loss_fn(model(x), y)6 scaler.scale(loss).backward()7 scaler.unscale_(opt)# so clipping sees true gradients8 torch.nn.utils.clip_grad_norm_(model.parameters(),1.0)9 scaler.step(opt)# skipped if grads contain inf or NaN10 scaler.update()
An evaluation script runs out of GPU memory partway through a long validation set, although a single batch fits easily. The loop does total_loss += loss_fn(model(x), y). Find the bug, then describe how you debug GPU memory in general.
MediumPerformance, Memory and torch.compile
Sample Answer
Without torch.no_grad(), every forward records a graph and saves activations for backward. Because loss belongs to that graph, total_loss += loss turns total_loss into a node that references every batch's graph, so no activation is ever freed. Run evaluation under torch.inference_mode() and accumulate loss.item(), or loss.detach() if you want to avoid a device sync per batch. In general, compare torch.cuda.memory_allocated() (bytes held by live tensors) with torch.cuda.memory_reserved() (bytes held by the caching allocator), read torch.cuda.memory_summary(), and record torch.cuda.memory._record_memory_history() to get a snapshot showing which stack trace allocated each block. torch.cuda.empty_cache() never fixes a leak: it only releases cached blocks that no tensor uses.
You want to fine-tune a 7B-parameter model with AdamW on 80 GB GPUs using autocast. Estimate the memory, say whether it fits on one GPU, and list what you would do if it does not.
HardPerformance, Memory and torch.compile
Sample Answer
Count bytes per parameter. Under autocast the parameters stay float32 (4 bytes), their gradients are float32 (4 bytes), and AdamW keeps two float32 moment buffers (8 bytes). That is 16 bytes per parameter, so 7B parameters need about 112 GB before a single activation, which does not fit on one 80 GB GPU. Activations come on top and grow with batch size and sequence length. Options, roughly in order: shard parameters, gradients and optimizer state across GPUs with FSDP; recompute activations in backward with torch.utils.checkpoint.checkpoint(fn, x, use_reentrant=False); shrink the micro-batch and use gradient accumulation; or fine-tune a small adapter such as LoRA so optimizer state exists only for the adapter weights.
Python
params =7e9bytes_per_param =4+4+8# fp32 weights + fp32 grads + two AdamW momentsprint(params * bytes_per_param /1e9,"GB before activations")# 112.0
You wrap a model in torch.compile and see almost no speedup, and the first few batches are very slow. How do you find out why, and what typically causes graph breaks and recompiles?
HardPerformance, Memory and torch.compile
Sample Answer
The first calls are slow because Dynamo traces the Python code and Inductor generates kernels, so benchmark only after warmup. A small speedup usually means graph breaks: Dynamo meets something it cannot trace, ends the graph, runs that part in plain Python and starts a new graph. Common causes are .item() and Python branches on tensor values, print, NumPy calls and unsupported third-party code. Run with TORCH_LOGS="graph_breaks,recompiles" or torch._dynamo.explain(fn)(x) to list them, and fullgraph=True turns the first break into an error. Recompiles come from failed guards; by default a new input shape triggers one recompile with dynamic shapes, and torch._dynamo.config.recompile_limit (8) caps them.
Python
1import torch
2deff(x):3 y = x.sin()4if y.sum().item()>0:# data-dependent Python branch5 y = y *26return y.cos()7ex = torch._dynamo.explain(f)(torch.randn(8))8print(ex.graph_count, ex.graph_break_count)# 2 1
A colleague times an 8192 x 8192 matrix multiply on the GPU with time.time() before and after the call and reports 0.05 ms. Why is that number wrong, and how should you time GPU code?
EasyPerformance, Memory and torch.compile
Sample Answer
CUDA kernel launches are asynchronous. The Python call returns as soon as the kernel is queued, so the timer measured the launch, not the work. Call torch.cuda.synchronize() before reading the clock, or record a pair of torch.cuda.Event(enable_timing=True) objects around the work and read start.elapsed_time(end) after synchronizing. Run warmup iterations first, since the first call pays for library initialization and memory allocation, and average over many runs. torch.utils.benchmark.Timer handles warmup and synchronization for you, and torch.profiler.profile breaks a whole training step into CPU and CUDA time per op.
Python
1import torch
2a = torch.randn(8192,8192, device="cuda")3for _ inrange(3):4 a @ a # warmup5start = torch.cuda.Event(enable_timing=True)6end = torch.cuda.Event(enable_timing=True)7start.record()8for _ inrange(10):9 a @ a
10end.record()11torch.cuda.synchronize()12print(start.elapsed_time(end)/10,"ms per matmul")
Once a model outgrows one GPU or has to serve traffic, the questions change. ML infrastructure and research engineering interviews check whether you know what each process does in data-parallel training and what a model needs before it can leave Python.
For distributed training, the rule is that every rank must issue the same collective operations in the same order, because a rank that skips one leaves the others waiting forever. For deployment, the goal is a full graph with no Python left in it.
Distributed Training and Deployment
Explain how DistributedDataParallel works and why it replaced nn.DataParallel. What do you change in data loading?
MediumDistributed Training and Deployment
Sample Answer
DDP runs one process per GPU, usually launched with torchrun, and each process holds a full model replica. Each process runs forward and backward on its own slice of the data. During backward, DDP all-reduces gradients in buckets (25 MiB by default, set by bucket_cap_mb) as soon as a bucket is ready, so communication overlaps with computation and every replica ends the step with the same averaged gradients. nn.DataParallel is a single process that scatters inputs and replicates the model on every step, and the GIL limits it, so it scales poorly. For data, use DistributedSampler and call sampler.set_epoch(epoch) at the start of each epoch, otherwise every epoch repeats the same shuffled order.
Python
1# launch: torchrun --nproc_per_node=8 train.py2import os, torch, torch.distributed as dist
3from torch.nn.parallel import DistributedDataParallel as DDP
4from torch.utils.data import DataLoader, DistributedSampler
5dist.init_process_group("nccl")6rank =int(os.environ["LOCAL_RANK"])7torch.cuda.set_device(rank)8model = DDP(build_model().cuda(rank), device_ids=[rank])9sampler = DistributedSampler(dataset)10loader = DataLoader(dataset, batch_size=32, sampler=sampler)11for epoch inrange(num_epochs):12 sampler.set_epoch(epoch)# new shuffle every epoch
A DDP job crashes on its second iteration with Expected to have finished reduction in the prior iteration before starting a new one. Another DDP job simply hangs after the first epoch. What causes each?
HardDistributed Training and Deployment
Sample Answer
The error means some parameters received no gradient in the previous step, usually a branch that the forward pass skipped, such as an auxiliary head used only on some batches. DDP's reducer waits for every parameter's gradient before it can finish the all-reduce. Pass find_unused_parameters=True, which costs an extra graph traversal per step, or make every parameter take part. If the set of unused parameters never changes, static_graph=True is cheaper. The hang is almost always a collective that not every rank calls: evaluation or checkpoint code under if rank == 0: that runs a DDP forward, all_reduce or barrier, or ranks with different numbers of batches. Every rank must issue the same collectives in the same order.
You add gradient accumulation to a DDP job and the step time barely improves. Why, and how do you fix it?
MediumDistributed Training and Deployment
Sample Answer
DDP all-reduces gradients during every backward call, so 4 micro-batches per optimizer step cost 4 all-reduces where 1 would do. Run the non-final micro-batches inside model.no_sync(). Gradients then accumulate locally in .grad, and the next backward outside the context all-reduces the accumulated total. Both the forward and the backward of those micro-batches must sit inside the context. Scale each loss by the accumulation count, as on one GPU. FSDP2 offers the same switch through set_requires_gradient_sync(False) on the sharded module, but skipping the sync there holds unsharded gradients, so it trades memory for communication.
Your model's parameters, gradients and AdamW state no longer fit on one GPU, so DDP is out. How does FSDP work, what do you wrap, and what changes for checkpointing?
HardDistributed Training and Deployment
Sample Answer
FSDP shards parameters, gradients and optimizer state across ranks, so each GPU stores about 1/N of them. Before a wrapped unit runs forward, FSDP all-gathers its full parameters, runs it and frees the unsharded copy. Backward gathers again, then reduce-scatters gradients so each rank keeps only its shard. Granularity matters: shard each transformer block and then the root module, so only one block is unsharded at a time and the next block's all-gather overlaps with compute. The current API is FSDP2, torch.distributed.fsdp.fully_shard, which stores shards as DTensor. Build the optimizer after sharding. For checkpoints, use torch.distributed.checkpoint so every rank writes its own shard in parallel instead of gathering the full model on rank 0.
Python
1import torch
2from torch.distributed.fsdp import fully_shard
3# inside a torchrun job, after init_process_group4for block in model.layers:5 fully_shard(block)# one all-gather unit per block6fully_shard(model)# root: embeddings, final norm, head7opt = torch.optim.AdamW(model.parameters(), lr=3e-4)# after sharding
You need to ship a PyTorch model to a runtime with no Python. What does torch.export produce, how do you allow a variable batch size, and which code patterns break it?
MediumDistributed Training and Deployment
Sample Answer
torch.export.export(model, args) traces the model ahead of time into an ExportedProgram: one graph of ATen operators plus its parameters, with no Python fallback. Unlike torch.compile, it cannot break the graph, so anything it cannot capture is an error. Shapes are specialized to the example inputs unless you mark them dynamic with dynamic_shapes and torch.export.Dim. A Python if on tensor values fails; rewrite it with torch.cond or tensor ops such as torch.where. The result can be saved with torch.export.save, compiled ahead of time with AOTInductor, or converted by torch.onnx.export, which now uses this path by default. torch.jit.script is deprecated, so new work should target torch.export.
Python
1import torch
2import torch.nn as nn
3from torch.export import export, Dim
4model = nn.Sequential(nn.Linear(4,8), nn.ReLU(), nn.Linear(8,2)).eval()5batch = Dim("batch")6ep = export(model,(torch.randn(3,4),), dynamic_shapes=({0: batch},))7torch.export.save(ep,"model.pt2")8print(ep.module()(torch.randn(10,4)).shape)# torch.Size([10, 2])
Two GPU training runs with the same seed diverge after a few hundred steps. What does it take to make PyTorch training reproducible, and what does it cost?
EasyDistributed Training and Deployment
Sample Answer
Seed everything: torch.manual_seed (which covers all devices), Python's random and NumPy, and pass a seeded generator to the DataLoader so shuffling repeats. Seeds alone are not enough. Some CUDA kernels are nondeterministic, for example scatter and index operations built on atomic adds, and cuDNN autotuning can choose different convolution algorithms per run. Set torch.backends.cudnn.benchmark = False and call torch.use_deterministic_algorithms(True), which raises an error when an op has no deterministic implementation. cuBLAS also needs the environment variable CUBLAS_WORKSPACE_CONFIG=:4096:8. Expect slower steps, and do not expect identical bits across GPU models or PyTorch versions.
For every reshape, transpose, slice and expand you write, say out loud whether it returns a view or a copy and what the strides will be, then check with .stride() and .data_ptr(). Half the tensor questions reduce to this habit. The lessons Storage, strides and views and Views, copies and contiguous memory drill it.
Write the training loop from a blank file
Interviewers often ask for a loop with train and eval modes, gradient clipping, accumulation and checkpointing, typed without notes. Practice it until it takes under ten minutes, then plant bugs in it and find them. The training loop and Gradient accumulation and clipping cover the details people forget.
Break autograd on purpose
Trigger each common error yourself: backward twice, an in-place op on a saved tensor, an inference tensor used in training, a leaf modified in place. Reading the error text once makes it easy to explain in an interview. Then write one custom autograd.Function and pass it through gradcheck, as in no_grad, inference_mode, detach and in-place ops and Custom autograd Functions.
Do the memory arithmetic by hand
Be ready to estimate bytes per parameter for weights, gradients and optimizer state, then say how activations scale with batch size and sequence length. Pair that with the AMP rules for float16 versus bfloat16. Where the memory goes and Mixed precision: autocast and bfloat16 walk through the numbers.
Read compiler and distributed logs
Run torch.compile with TORCH_LOGS="graph_breaks,recompiles" on a model you know, and launch a two-process DDP job on CPU with the gloo backend so you can reproduce hangs and unused-parameter errors without a cluster. See torch.compile and graph breaks and DDP and FSDP.
How Ready Are You for PyTorch Interviews?
1 / 6
Tensors
y = x.t() and then y.view(-1) raises an error. What is the best explanation and fix?
Frequently Asked Questions
Is PyTorch asked in data science interviews?
It depends on the role. Product and analytics data science loops rarely test PyTorch directly. Machine learning engineer, research engineer, applied scientist and deep learning focused data scientist roles often do, through a live coding round that asks for a model or training loop, or a debugging round built around a broken script. If the job description mentions deep learning, expect at least one question on autograd, training loops or GPU memory.
How long does it take to prepare for a PyTorch interview?
If you already train models in PyTorch at work, two to three weeks of focused practice is usually enough to cover internals such as strides, autograd, mixed precision and DDP. Coming from another framework, plan on four to six weeks, starting with tensors and the training loop before moving to performance and distributed training. Writing code daily matters more than reading, because interviewers watch you type and debug.
What level of PyTorch do I need for a machine learning engineer interview?
You should write a correct training and evaluation loop without notes, explain autograd well enough to debug in-place and graph errors, and reason about GPU memory and mixed precision. Senior and infrastructure roles add DDP, FSDP, torch.compile and profiling. Research roles lean toward implementing layers or losses from scratch with correct shapes and gradients. Model export matters most for roles that own serving.
Should I learn PyTorch or TensorFlow for ML interviews?
PyTorch, unless the job description names TensorFlow or JAX. Most research code and a large share of production training code is now written in PyTorch, and interviewers who let you pick a framework usually expect it. The concepts transfer: automatic differentiation, batching, mixed precision and data parallelism work the same way in every framework, so a strong PyTorch foundation also prepares you for a TensorFlow question.
Do PyTorch interviews include live coding?
Usually. Common formats are implementing a module such as multi-head attention or a custom loss, writing a training loop for a toy dataset, or finding the bugs in a script that trains but does not learn. Interviewers watch whether you check shapes, switch train and eval modes, and handle devices and dtypes. Some loops add a design discussion about scaling training across GPUs.
Which PyTorch version should I study for interviews in 2026?
Study current PyTorch 2.x behavior. Know that torch.compile is the main path to speedups, that torch.load defaults to weights_only=True since 2.6, that torch.amp.GradScaler replaces the older torch.cuda.amp version, that FSDP2's fully_shard is the recommended FSDP API, and that torch.export has replaced TorchScript for new deployment work. Interviewers notice when answers describe deprecated behavior.
Written by
Dan Lee
Data & AI Lead
Dan is a seasoned data scientist and ML coach with 10+ years of experience at Google, PayPal, and startups. He has helped candidates land top-paying roles and offers personalized guidance to accelerate your data career.