PyTorch Interview Questions

Dan Lee's profile image
Dan LeeData & AI Lead
Last update
PyTorch Interview Questions

PyTorch interviews rarely ask you to define a tensor. They hand you a training loop that runs but learns nothing, a job that runs out of memory at step 2,000, or a DDP launch that hangs, and ask you to find the cause. Machine learning engineers, research engineers and applied scientists get these questions because the bugs live in the details of how PyTorch stores data and records gradients.

Candidates usually slip on the mechanics under the API. Which operations return views? What does model.eval() leave unchanged? Why does torch.load now refuse some checkpoints, and where does GPU memory actually go? A strong answer names the fix and then explains what PyTorch is doing underneath.

The 30 questions below are grouped into five areas: tensors, autograd, training loops, performance, and distributed training with deployment. Each solution gives the answer, the reason and the trap, with short code checked against current PyTorch 2.x. A preparation plan, a quiz and an FAQ follow the questions.

Intermediate30 questions

PyTorch Interview Questions

Practice questions on how PyTorch stores tensors, computes gradients and trains at scale, with answers checked against current PyTorch 2.x.

Machine Learning EngineerResearch EngineerApplied ScientistAI EngineerML Infrastructure EngineerData Scientist

Tensors, Strides and Views

Many PyTorch bugs that survive code review are shape and memory bugs. Interviewers use tensor questions to find out whether you know what a tensor is underneath the API: a storage buffer plus a shape, one stride per dimension and an offset.

The idea that matters most is that many operations return views sharing storage with their input. Once you can predict which calls alias and which ones copy, errors like view size is not compatible and silent in-place corruption stop being mysterious.

Tensors, Strides and Views

You transpose a (3, 4) tensor and call y.view(-1) to flatten it before a linear layer. PyTorch raises view size is not compatible with input tensor's size and stride. What happened, and how do you fix it?

EasyTensors, Strides and Views

Sample Answer

torch.arange(12).reshape(3, 4) has strides (4, 1): one step along a row moves 1 element in memory, one step down a column moves 4. .t() copies nothing. It swaps the strides to (1, 4), so the tensor is no longer contiguous in row-major order. view can only reinterpret shape and strides over the same memory, and no single stride walks the transposed elements in order, so it fails. Use y.reshape(-1), which returns a view when the strides allow it and copies otherwise, or y.contiguous().view(-1) to make the copy explicit. The trap is assuming reshape always aliases. After a copy, in-place writes to the result no longer reach the original.

Python
1import torch
2x = torch.arange(12).reshape(3, 4)
3y = x.t()
4print(x.stride(), y.stride(), y.is_contiguous())  # (4, 1) (1, 4) False
5flat = y.reshape(-1)                              # copies here
6print(flat.data_ptr() == x.data_ptr())            # False
Learn this in the PyTorch course

Autograd and Gradients

Autograd questions separate people who call loss.backward() from people who can debug it. Expect errors raised in the backward pass, gradients that come out as None or zero, and prompts that ask you to write your own backward.

The core idea: during the forward pass PyTorch records a graph and saves the tensors each backward formula needs. Most autograd errors are about that saved state. It was freed, it was modified in place, or it was never recorded.

Autograd and Gradients

You compute two losses from one forward pass and call loss1.backward() and then loss2.backward(). The second call raises Trying to backward through the graph a second time. Why, and what is the right fix?

EasyAutograd and Gradients

Sample Answer

After a backward pass, autograd frees the intermediate tensors it saved for computing gradients, to reclaim memory. The second call reaches the shared part of the graph and finds those buffers gone. The usual fix is (loss1 + loss2).backward(): one traversal, the same gradients. loss1.backward(retain_graph=True) also works, but it keeps the saved tensors alive and costs memory. Also remember that gradients accumulate. Two backward calls add into .grad rather than overwrite it, which is why training loops zero gradients every step and why gradient accumulation works at all.

Python
1import torch
2x = torch.randn(3, requires_grad=True)
3h = x.exp()
4loss1, loss2 = h.sum(), (2 * h).sum()
5(loss1 + loss2).backward()   # one pass through the shared graph
6print(x.grad)                # equals 3 * exp(x)
Learn this in the PyTorch course

nn.Module and Training Loops

Every PyTorch role expects you to write a training loop from memory and to spot what is wrong with someone else's. The bugs interviewers plant are small: a missing zero_grad, a model left in training mode, a tensor that never reached the GPU.

What matters most is the state a module carries besides its weights, such as the training flag, buffers like BatchNorm's running statistics, and which attributes are registered at all. Most loop bugs come from state you forgot was there.

nn.Module and Training Loops

Validation accuracy changes every time you evaluate the same checkpoint on the same data. What is the most likely cause, and what does model.eval() change?

Easynn.Module and Training Loops

Sample Answer

The model is still in training mode. There, nn.Dropout zeroes activations at random (p=0.5 by default) and scales the rest by 1/(1-p), so predictions vary run to run. nn.BatchNorm normalizes with the current batch's statistics and updates its running mean and variance with momentum=0.1, so evaluating also changes the model. model.eval() sets training=False recursively: Dropout becomes the identity and BatchNorm uses its stored running statistics. It does not disable gradient tracking, so pair it with torch.no_grad() or torch.inference_mode(), and call model.train() before the next training epoch.

Learn this in the PyTorch course

Performance, Memory and torch.compile

Senior ML engineering interviews spend real time on why a job is slow or out of memory. Interviewers want a method: measure, name the bottleneck, then choose the fix that targets it.

Keep the budgets separate. A training step spends time in the input pipeline and in GPU compute, and its memory splits into weights, gradients, optimizer state and activations. Each fix in this section moves one of those numbers.

Performance, Memory and torch.compile

GPU utilization hovers around 30% while training an image model, and the GPU idles between bursts. How do you confirm the input pipeline is the bottleneck, and which DataLoader settings do you change?

MediumPerformance, Memory and torch.compile

Sample Answer

Measure first: time an epoch that only iterates the DataLoader, or profile a few steps with torch.profiler and look for gaps where the GPU waits for the next batch. If loading is the bottleneck, raise num_workers from its default of 0, which loads on the main process, toward the CPU cores available per GPU. Set pin_memory=True and call .to(device, non_blocking=True) so host-to-device copies run asynchronously. persistent_workers=True avoids restarting workers every epoch, and prefetch_factor (2 batches per worker by default once workers exist) sets read-ahead. If decoding or augmentation is still too slow, move it to the GPU. PyTorch seeds torch, random and NumPy differently in each worker, but other libraries need a worker_init_fn.

Learn this in the PyTorch course

Distributed Training and Deployment

Once a model outgrows one GPU or has to serve traffic, the questions change. ML infrastructure and research engineering interviews check whether you know what each process does in data-parallel training and what a model needs before it can leave Python.

For distributed training, the rule is that every rank must issue the same collective operations in the same order, because a rank that skips one leaves the others waiting forever. For deployment, the goal is a full graph with no Python left in it.

Distributed Training and Deployment

Explain how DistributedDataParallel works and why it replaced nn.DataParallel. What do you change in data loading?

MediumDistributed Training and Deployment

Sample Answer

DDP runs one process per GPU, usually launched with torchrun, and each process holds a full model replica. Each process runs forward and backward on its own slice of the data. During backward, DDP all-reduces gradients in buckets (25 MiB by default, set by bucket_cap_mb) as soon as a bucket is ready, so communication overlaps with computation and every replica ends the step with the same averaged gradients. nn.DataParallel is a single process that scatters inputs and replicates the model on every step, and the GIL limits it, so it scales poorly. For data, use DistributedSampler and call sampler.set_epoch(epoch) at the start of each epoch, otherwise every epoch repeats the same shuffled order.

Python
1# launch: torchrun --nproc_per_node=8 train.py
2import os, torch, torch.distributed as dist
3from torch.nn.parallel import DistributedDataParallel as DDP
4from torch.utils.data import DataLoader, DistributedSampler
5dist.init_process_group("nccl")
6rank = int(os.environ["LOCAL_RANK"])
7torch.cuda.set_device(rank)
8model = DDP(build_model().cuda(rank), device_ids=[rank])
9sampler = DistributedSampler(dataset)
10loader = DataLoader(dataset, batch_size=32, sampler=sampler)
11for epoch in range(num_epochs):
12    sampler.set_epoch(epoch)        # new shuffle every epoch
Learn this in the PyTorch course

How to Prepare for PyTorch Interviews

Predict strides before you run the code

For every reshape, transpose, slice and expand you write, say out loud whether it returns a view or a copy and what the strides will be, then check with .stride() and .data_ptr(). Half the tensor questions reduce to this habit. The lessons Storage, strides and views and Views, copies and contiguous memory drill it.

Write the training loop from a blank file

Interviewers often ask for a loop with train and eval modes, gradient clipping, accumulation and checkpointing, typed without notes. Practice it until it takes under ten minutes, then plant bugs in it and find them. The training loop and Gradient accumulation and clipping cover the details people forget.

Break autograd on purpose

Trigger each common error yourself: backward twice, an in-place op on a saved tensor, an inference tensor used in training, a leaf modified in place. Reading the error text once makes it easy to explain in an interview. Then write one custom autograd.Function and pass it through gradcheck, as in no_grad, inference_mode, detach and in-place ops and Custom autograd Functions.

Do the memory arithmetic by hand

Be ready to estimate bytes per parameter for weights, gradients and optimizer state, then say how activations scale with batch size and sequence length. Pair that with the AMP rules for float16 versus bfloat16. Where the memory goes and Mixed precision: autocast and bfloat16 walk through the numbers.

Read compiler and distributed logs

Run torch.compile with TORCH_LOGS="graph_breaks,recompiles" on a model you know, and launch a two-process DDP job on CPU with the gloo backend so you can reproduce hangs and unused-parameter errors without a cluster. See torch.compile and graph breaks and DDP and FSDP.

How Ready Are You for PyTorch Interviews?

1 / 6
Tensors

y = x.t() and then y.view(-1) raises an error. What is the best explanation and fix?

Frequently Asked Questions

Is PyTorch asked in data science interviews?

It depends on the role. Product and analytics data science loops rarely test PyTorch directly. Machine learning engineer, research engineer, applied scientist and deep learning focused data scientist roles often do, through a live coding round that asks for a model or training loop, or a debugging round built around a broken script. If the job description mentions deep learning, expect at least one question on autograd, training loops or GPU memory.

How long does it take to prepare for a PyTorch interview?

If you already train models in PyTorch at work, two to three weeks of focused practice is usually enough to cover internals such as strides, autograd, mixed precision and DDP. Coming from another framework, plan on four to six weeks, starting with tensors and the training loop before moving to performance and distributed training. Writing code daily matters more than reading, because interviewers watch you type and debug.

What level of PyTorch do I need for a machine learning engineer interview?

You should write a correct training and evaluation loop without notes, explain autograd well enough to debug in-place and graph errors, and reason about GPU memory and mixed precision. Senior and infrastructure roles add DDP, FSDP, torch.compile and profiling. Research roles lean toward implementing layers or losses from scratch with correct shapes and gradients. Model export matters most for roles that own serving.

Should I learn PyTorch or TensorFlow for ML interviews?

PyTorch, unless the job description names TensorFlow or JAX. Most research code and a large share of production training code is now written in PyTorch, and interviewers who let you pick a framework usually expect it. The concepts transfer: automatic differentiation, batching, mixed precision and data parallelism work the same way in every framework, so a strong PyTorch foundation also prepares you for a TensorFlow question.

Do PyTorch interviews include live coding?

Usually. Common formats are implementing a module such as multi-head attention or a custom loss, writing a training loop for a toy dataset, or finding the bugs in a script that trains but does not learn. Interviewers watch whether you check shapes, switch train and eval modes, and handle devices and dtypes. Some loops add a design discussion about scaling training across GPUs.

Which PyTorch version should I study for interviews in 2026?

Study current PyTorch 2.x behavior. Know that torch.compile is the main path to speedups, that torch.load defaults to weights_only=True since 2.6, that torch.amp.GradScaler replaces the older torch.cuda.amp version, that FSDP2's fully_shard is the recommended FSDP API, and that torch.export has replaced TorchScript for new deployment work. Interviewers notice when answers describe deprecated behavior.

Dan Lee's profile image

Written by

Dan Lee

Data & AI Lead

Dan is a seasoned data scientist and ML coach with 10+ years of experience at Google, PayPal, and startups. He has helped candidates land top-paying roles and offers personalized guidance to accelerate your data career.

Connect on LinkedIn