CUDA Interview Questions

Dan Lee's profile image
Dan LeeData & AI Lead
Last update
CUDA Interview Questions

CUDA interviews for GPU kernel, ML infrastructure and inference roles rarely ask you to define a kernel. They hand you an indexing expression, a profiler number or a launch configuration and ask what the hardware will do. How many sectors does this load touch? Which budget caps occupancy? Why did adding one cudaMemset serialize a pipeline? Good answers come with numbers, and the numbers depend on the architecture, so name the GPU you are reasoning about.

Candidates slip in predictable places. They chase 100% occupancy, put __syncthreads() inside a branch, assume a warp runs in lockstep, call FlashAttention approximate, or time an asynchronous launch with a CPU clock. Strong answers start from a floor (bytes over bandwidth, FLOPs over peak), explain the mechanism, and name the tool that proves it: Nsight Systems, Nsight Compute or compute-sanitizer.

The 31 questions below are grouped into five areas: the execution model, the memory hierarchy, synchronization, performance analysis with streams, and the kernels behind deep learning. Numbers are stated for a specific GPU (mostly A100 and H100) and reflect the CUDA 13.x toolkits. We could not compile the snippets on a GPU, so they were syntax-checked with Clang's CUDA front end and checked against NVIDIA's programming and best practices guides. The page ends with prep advice, a short quiz and an FAQ.

Advanced31 questions

CUDA Interview Questions

Practice CUDA interview questions on warps, occupancy, coalescing, shared memory, synchronization, profiling, streams, Tensor Cores and FlashAttention, with worked answers and kernel code.

GPU Kernel EngineerMachine Learning Infrastructure EngineerInference EngineerHigh Performance Computing EngineerResearch Engineer

Execution Model: Warps, Divergence and Occupancy

Every CUDA interview starts by checking that you think in warps rather than threads. Launch shapes, branch costs, occupancy and tail effects all follow from one hardware fact, and interviewers use them to see whether you can predict performance before you profile.

The idea that matters most: a block is cut into warps of 32 consecutive threads, and each warp issues one instruction at a time for all of its active lanes. An SM hides memory latency by switching between resident warps, so how many warps fit (and how much independent work each one has) decides how busy the hardware stays.

Execution Model: Warps, Divergence and Occupancy

You need to scale and add two arrays of 10 million floats (y = a * x + y). Write the kernel and the launch, and explain how you chose the block and grid sizes.

EasyExecution Model: Warps, Divergence and Occupancy

Sample Answer

Use a block size that is a multiple of 32, because warps are 32 threads and a partial last warp still takes a full warp slot. The Best Practices Guide suggests 128 to 256 threads as a starting range; the hard limit is 1,024 per block. For one thread per element the grid is ceil(n / 256), which is 39,063 blocks here, and the last block needs a bounds check. A grid-stride loop is more robust: it works for any n, keeps neighbouring lanes on neighbouring addresses so loads stay coalesced, and lets you size the grid to the GPU (a few blocks per SM) instead of to the data. The trap at very large sizes is an int index overflowing past 2^31 elements; switch to size_t there.

C++
1__global__ void saxpy(int n, float a, const float* __restrict__ x, float* __restrict__ y) {
2    for (int i = blockIdx.x * blockDim.x + threadIdx.x; i < n; i += gridDim.x * blockDim.x)
3        y[i] = a * x[i] + y[i];
4}
5
6void launch_saxpy(int n, float a, const float* d_x, float* d_y, cudaStream_t s) {
7    int threads = 256;
8    int blocks = (n + threads - 1) / threads;   // 39,063 for n = 10,000,000
9    saxpy<<<blocks, threads, 0, s>>>(n, a, d_x, d_y);
10}
Learn this in the GPU Programming course

Memory Hierarchy, Coalescing and Bank Conflicts

Most real kernels are limited by memory, not math, so memory questions take up a large share of any CUDA loop. Interviewers hand you an indexing expression or a profiler number and ask what the hardware fetches, where the data lives, and why the kernel is slow.

The idea that matters most: global memory moves in 32-byte sectors, and a warp-wide load costs the number of distinct sectors its 32 addresses touch. Shared memory has a similar rule with 32 banks of 4 bytes. Almost every fix in this section is a change of layout or index mapping that makes neighbouring lanes touch neighbouring words.

Memory Hierarchy, Coalescing and Bank Conflicts

Particles are stored as an array of struct { float x, y, z, mass; }. A hot kernel reads only x, one particle per thread, and runs at a quarter of the bandwidth you expected. Why, and what do you change?

EasyMemory Hierarchy, Coalescing and Bank Conflicts

Sample Answer

Lane i reads float 4 * i, a 16-byte stride. Global memory is served in 32-byte sectors, so a warp's 32 reads span 512 bytes and touch 16 sectors, of which it uses 128 bytes: 25% efficiency. A fully coalesced float load touches 4 sectors, and Nsight Compute reports this as sectors per request, 4 ideal against 16 here. Store the hot fields as a struct of arrays (all x values contiguous) and the same read becomes perfectly coalesced. The nuance: if the kernel uses all four fields, the array of structs is fine when each thread loads its particle as one aligned float4, a single 16-byte vector load per lane that touches the same 16 sectors and uses all of them.

Learn this in the GPU Programming course

Synchronization, Atomics and Warp-Level Primitives

Synchronization bugs are the CUDA bugs that pass tests and fail in production: a missing barrier, a barrier inside a branch, an atomic on a hot address, a shuffle mask that names lanes that are not there. Interviewers use them to check whether you know exactly what each primitive guarantees and what it costs.

The idea that matters most: CUDA gives you cheap synchronization at three scopes. A warp synchronizes through _sync intrinsics and __syncwarp(), a block through __syncthreads(), and a grid only through a kernel boundary or a cooperative launch. Pick the narrowest scope that makes the data dependency safe.

Synchronization, Atomics and Warp-Level Primitives

A block reduction loop does if (tid < s) { buf[tid] += buf[tid + s]; __syncthreads(); }. It hangs on some GPUs and gives wrong sums on others. Explain and fix it.

EasySynchronization, Atomics and Warp-Level Primitives

Sample Answer

__syncthreads() may appear in conditional code only if the condition evaluates the same way for every thread in the block. Here threads with tid >= s skip the barrier, so the behaviour is undefined: it can hang, or threads can read partial sums before they are written. Move the barrier out of the branch so every thread reaches it on every iteration, and keep only the work inside the condition. A related detail: the barrier waits for all non-exited threads, so an early return for out-of-range threads does not deadlock it, but those threads never write their shared slot, so load the identity value (0 for a sum) instead of returning. compute-sanitizer --tool synccheck reports this class of bug.

C++
1for (int s = blockDim.x / 2; s > 0; s >>= 1) {
2    if (tid < s) buf[tid] += buf[tid + s];
3    __syncthreads();          // every thread, every iteration
4}
Learn this in the GPU Programming course

Performance Analysis, Streams and Concurrency

Performance rounds test whether you can say how fast a kernel should be before you measure it, then measure it honestly. Expect a roofline estimate, a profiler report to interpret, a timing bug to spot, and a host-side pipeline that fails to overlap.

The idea that matters most: every kernel has a floor, the larger of FLOPs divided by peak math and bytes divided by peak bandwidth. Compare the measured time with that floor first. If the gap is small, stop optimizing the kernel; if the GPU is idle between kernels, the problem is on the host, and streams, pinned memory or CUDA graphs are the fix.

Performance Analysis, Streams and Concurrency

A vector add of 100 million floats takes 700 µs on an A100 80GB. Is it memory-bound or compute-bound, and is there anything left to optimize inside the kernel?

EasyPerformance Analysis, Streams and Concurrency

Sample Answer

It reads two floats and writes one per element: 12 bytes for 1 FLOP, an arithmetic intensity of about 0.083 FLOP/byte. The A100's FP32 ridge point is 19.5 TFLOPS ÷ about 2.0 TB/s, roughly 9.75 FLOP/byte, so the kernel sits far left of the ridge and is memory-bound. Its floors: 1.2 GB ÷ 2.0 TB/s is 600 µs for memory, against about 5 µs for math. At 700 µs it already achieves about 86% of peak bandwidth (effective bandwidth is bytes read plus bytes written over time). Faster math changes nothing. The only real gains come from moving fewer bytes: fuse it with the kernels before and after so the intermediate never touches HBM, or store the data in BF16 or FP16 to halve the traffic.

Learn this in the GPU Programming course

Kernels in Practice and the ML Angle

ML infrastructure and inference roles ask you to write or explain the kernels that dominate model time: matrix multiply, softmax, attention and quantized matrix-vector products. They also want to know when you would write CUDA C++ at all, as opposed to Triton or a library, and how you debug a kernel that gives wrong answers.

The idea that matters most: deep learning kernels are fast when data is reused on chip and intermediate results never round-trip through HBM. Tiling, fusion, FlashAttention and fused dequantization are all the same move: raise arithmetic intensity by keeping bytes in registers and shared memory, then use Tensor Cores for whatever math is left.

Kernels in Practice and the ML Angle

Write a shared-memory tiled matrix multiply for square FP32 matrices of any size, and explain what tiling buys over the naive kernel.

MediumKernels in Practice and the ML Angle

Sample Answer

The naive kernel gives each thread one output and reads a full row of A and column of B from global memory: two 4-byte loads per multiply-add, about 0.25 FLOP per requested byte. Tiling has a block of TILE × TILE threads load one tile of A and one of B into shared memory, then each thread does TILE multiply-adds from shared memory, so each global load serves TILE operations: intensity TILE ÷ 4, so 4 FLOP/byte at TILE 16. Two barriers per phase: one after the loads so tiles are complete, one after the compute so no warp overwrites a tile another warp is still reading. Out-of-range threads load zeros but still reach both barriers. Next come register tiling and Tensor Cores; production code calls cuBLAS or CUTLASS.

C++
1#define T 16
2__global__ void matmul(const float* A, const float* B, float* C, int N) {
3    __shared__ float As[T][T], Bs[T][T];
4    int r = blockIdx.y * T + threadIdx.y, c = blockIdx.x * T + threadIdx.x;
5    float acc = 0.0f;
6    for (int k0 = 0; k0 < N; k0 += T) {
7        As[threadIdx.y][threadIdx.x] = (r < N && k0 + threadIdx.x < N) ? A[r * N + k0 + threadIdx.x] : 0.0f;
8        Bs[threadIdx.y][threadIdx.x] = (c < N && k0 + threadIdx.y < N) ? B[(k0 + threadIdx.y) * N + c] : 0.0f;
9        __syncthreads();
10        for (int k = 0; k < T; ++k) acc += As[threadIdx.y][k] * Bs[k][threadIdx.x];
11        __syncthreads();
12    }
13    if (r < N && c < N) C[r * N + c] = acc;
14}
Learn this in the GPU Programming course

How to Prepare for CUDA Interviews

Do the occupancy arithmetic by hand

Know the per-SM limits of at least one GPU from memory (an A100 has 64 warps, 32 blocks, 65,536 registers and up to 164 KB of shared memory per SM) and practise turning a launch configuration into resident blocks. Then be ready to argue why low occupancy can be fine. The Occupancy lesson walks through the calculation and the Little's law argument.

Count sectors and banks for any index

Given in[f(threadIdx.x)], you should be able to say how many 32-byte sectors a warp touches and, for shared memory, how many ways it conflicts. Work through Coalescing and Shared memory and bank conflicts until the padding trick is automatic.

Write the classic kernels without notes

A block reduction with warp shuffles, a privatized histogram, a tiled matmul and a fused softmax cover most live-coding prompts. Write each one, then explain where every barrier goes and why. The Reduction and Kernel round lessons build them step by step.

Open every performance answer with a floor

Before discussing fixes, compute FLOPs over peak and bytes over bandwidth for the GPU in question, then compare with the measured time. Interviewers notice when a candidate knows a kernel is already at 86% of bandwidth. The roofline model and Profiling with Nsight Compute show how to get both numbers and read a report in order.

Connect kernels to model-level costs

For ML and inference roles, practise explaining FlashAttention with byte counts, why batch-1 decode is memory-bound, and when launch overhead dominates a step. The FlashAttention lesson and Fusion, CUDA graphs and launch overhead cover these with real numbers.

How Ready Are You for CUDA Interviews?

1 / 6
Execution Model

A kernel branches on threadIdx.x % 2, with 20 instructions on each path. What does each warp pay?

Frequently Asked Questions

Is CUDA asked in machine learning engineer interviews?

It depends on the team. Research and product ML roles rarely test CUDA directly, though they may ask why a model is memory-bound or how mixed precision works. ML infrastructure, inference, compiler and GPU kernel roles do test it, often with a live kernel (a reduction, softmax or tiled matmul) and a performance discussion built on the roofline model and profiler output. If the job description mentions kernels, Triton, TensorRT or GPU performance, prepare for CUDA questions.

Which CUDA version should I know for interviews in 2026?

Interviews test concepts that have been stable for years: warps, the memory hierarchy, barriers, streams. Still, know the current context. CUDA 13 toolkits dropped offline compilation for GPUs older than compute capability 7.5, so every target has independent thread scheduling and you should use the _sync warp intrinsics. Be able to name what Hopper and Blackwell added (thread block clusters, TMA, FP8, FP4 Tensor Cores) at a high level, even if you mostly work through libraries.

How long does it take to prepare for a CUDA interview?

If you already write C++ and know basic GPU concepts, three to four weeks of steady practice is usually enough: one week on the execution model and memory, one on writing the classic kernels, one on profiling and streams, and time for ML kernels if the role needs them. Starting from scratch, plan on two to three months. Writing kernels and checking them against a reference teaches far more than reading.

Do I need access to a GPU to prepare?

It helps but is not required. Much of interview reasoning is arithmetic: sectors per request, bank conflicts, occupancy, roofline floors. You can practise that on paper or in a simulator. For timing and profiling, a cloud GPU instance for a few hours is enough to run Nsight Compute on your own kernels. Know the commands and how to read the report even if you profile only a handful of times.

Should I learn Triton or CUDA C++ first?

For ML roles, learn the CUDA execution and memory model first, because Triton answers still rely on it: coalescing, occupancy and the roofline do not go away. Then write kernels in whichever language the team uses. Triton gets you to fused kernels quickly and is what torch.compile emits. CUDA C++ remains the language of interview whiteboards, CUTLASS and inference engines, so be able to read and write a simple kernel in it either way.

What level of CUDA knowledge do inference engineer interviews expect?

Expect to explain why decode is memory-bound and prefill is compute-bound, what FlashAttention, paged attention and CUDA graphs fix, and how quantization changes the bytes moved per token. You should be able to read an Nsight Systems timeline and an Nsight Compute report, write a reduction or GEMV kernel, and estimate step time from model size and bandwidth. Deep Tensor Core programming is usually left to specialist kernel roles.

Dan Lee's profile image

Written by

Dan Lee

Data & AI Lead

Dan is a seasoned data scientist and ML coach with 10+ years of experience at Google, PayPal, and startups. He has helped candidates land top-paying roles and offers personalized guidance to accelerate your data career.

Connect on LinkedIn