CUDA interviews for GPU kernel, ML infrastructure and inference roles rarely ask you to define a kernel. They hand you an indexing expression, a profiler number or a launch configuration and ask what the hardware will do. How many sectors does this load touch? Which budget caps occupancy? Why did adding one cudaMemset serialize a pipeline? Good answers come with numbers, and the numbers depend on the architecture, so name the GPU you are reasoning about.
Candidates slip in predictable places. They chase 100% occupancy, put __syncthreads() inside a branch, assume a warp runs in lockstep, call FlashAttention approximate, or time an asynchronous launch with a CPU clock. Strong answers start from a floor (bytes over bandwidth, FLOPs over peak), explain the mechanism, and name the tool that proves it: Nsight Systems, Nsight Compute or compute-sanitizer.
The 31 questions below are grouped into five areas: the execution model, the memory hierarchy, synchronization, performance analysis with streams, and the kernels behind deep learning. Numbers are stated for a specific GPU (mostly A100 and H100) and reflect the CUDA 13.x toolkits. We could not compile the snippets on a GPU, so they were syntax-checked with Clang's CUDA front end and checked against NVIDIA's programming and best practices guides. The page ends with prep advice, a short quiz and an FAQ.
Advanced31 questions
CUDA Interview Questions
Practice CUDA interview questions on warps, occupancy, coalescing, shared memory, synchronization, profiling, streams, Tensor Cores and FlashAttention, with worked answers and kernel code.
Every CUDA interview starts by checking that you think in warps rather than threads. Launch shapes, branch costs, occupancy and tail effects all follow from one hardware fact, and interviewers use them to see whether you can predict performance before you profile.
The idea that matters most: a block is cut into warps of 32 consecutive threads, and each warp issues one instruction at a time for all of its active lanes. An SM hides memory latency by switching between resident warps, so how many warps fit (and how much independent work each one has) decides how busy the hardware stays.
Execution Model: Warps, Divergence and Occupancy
You need to scale and add two arrays of 10 million floats (y = a * x + y). Write the kernel and the launch, and explain how you chose the block and grid sizes.
EasyExecution Model: Warps, Divergence and Occupancy
Sample Answer
Use a block size that is a multiple of 32, because warps are 32 threads and a partial last warp still takes a full warp slot. The Best Practices Guide suggests 128 to 256 threads as a starting range; the hard limit is 1,024 per block. For one thread per element the grid is ceil(n / 256), which is 39,063 blocks here, and the last block needs a bounds check. A grid-stride loop is more robust: it works for any n, keeps neighbouring lanes on neighbouring addresses so loads stay coalesced, and lets you size the grid to the GPU (a few blocks per SM) instead of to the data. The trap at very large sizes is an int index overflowing past 2^31 elements; switch to size_t there.
C++
1__global__ voidsaxpy(int n,float a,constfloat* __restrict__ x,float* __restrict__ y){2for(int i = blockIdx.x * blockDim.x + threadIdx.x; i < n; i += gridDim.x * blockDim.x)3 y[i]= a * x[i]+ y[i];4}56voidlaunch_saxpy(int n,float a,constfloat* d_x,float* d_y, cudaStream_t s){7int threads =256;8int blocks =(n + threads -1)/ threads;// 39,063 for n = 10,000,0009 saxpy<<<blocks, threads,0, s>>>(n, a, d_x, d_y);10}
In code review you see two kernels that split work in half. One branches on threadIdx.x % 2 == 0, the other on (threadIdx.x / 32) % 2 == 0. Both paths are about 20 instructions long. Which is slower, and why?
EasyExecution Model: Warps, Divergence and Occupancy
Sample Answer
The % 2 version. A warp issues one instruction for all its active lanes, and when lanes disagree, it runs both paths with the other lanes masked off. Every warp has both even and odd lanes, so every warp pays about 40 issue slots, not 20. The / 32 version branches on the warp number, which is the same for all 32 lanes, so it is warp-uniform and costs nothing extra. A bounds check if (i < n) splits at most one warp, so it is harmless. A rare branch is not cheap, though. If 1% of elements take a slow path at random, about 28% of warps contain one and pay for it. Nsight Compute shows it as fewer than 32 active threads per instruction.
A kernel on an A100 (compute capability 8.0) uses 256 threads per block, 72 registers per thread and 8 KB of shared memory per block. What is its theoretical occupancy? A teammate proposes __launch_bounds__(256, 4) to raise it. What happens?
MediumExecution Model: Warps, Divergence and Occupancy
Sample Answer
Check every per-SM budget on cc 8.0 and take the smallest whole number of blocks. Warp slots: 64 warps, 8 per block, so 8 blocks. Block slots: 32. Shared memory: 164 KB per SM, and each block costs 8 KB plus 1 KB reserved by the runtime, so 18 blocks. Registers: 72 × 32 = 2,304 per warp, and 65,536 ÷ 2,304 is 28 warps, which is 3 whole blocks. Registers win: 3 blocks, 24 warps, 37.5%. __launch_bounds__(256, 4) asks for 4 resident blocks, so the compiler must fit in 65,536 ÷ 1,024 = 64 registers per thread, giving 50%. The trap is spilling: if the code really needs 72, the extra values go to local memory. Check --ptxas-options=-v for spill bytes and time both versions.
Nsight Compute shows your hand-written FP32 matmul at 25% achieved occupancy while it reaches about 70% of FP32 peak. Your manager asks you to raise occupancy. How do you respond?
MediumExecution Model: Warps, Divergence and Occupancy
Sample Answer
Occupancy is a means of hiding latency, and this kernel already hides it. What matters is enough independent work in flight, from many warps or from instruction-level parallelism inside each warp. Little's law gives the size: bytes in flight = bandwidth × latency. On an A100, about 2.0 TB/s × 500 ns is roughly 1 MB across the GPU, about 9.3 KB per SM. With one 4-byte load per thread you would need about 73 warps per SM, more than the 64 that fit. With four independent loads per thread, about 19 warps (30% occupancy) do the job. Tuned GEMMs hold a tile of outputs in registers per thread and run at low occupancy on purpose. Forcing occupancy up here would cap registers and likely cause spills.
On an H100 SXM (132 SMs), a kernel launches 140 blocks and only one block fits per SM because of its shared memory use. Profiling shows it takes about as long as a launch with 264 blocks. Explain and fix it.
MediumExecution Model: Warps, Divergence and Occupancy
Sample Answer
Blocks run in waves. With one resident block per SM, a wave is 132 blocks, so 140 blocks need two waves: the first runs 132 blocks, the second runs 8 while 124 SMs sit idle. With equal block times, 140 blocks cost two full waves, the same as 264, and the GPU is about 53% busy. Fixes: pick a tile size so the block count is close to a multiple of SMs × blocks per SM, shrink tiles so two blocks fit per SM, or split the work more finely. GEMM libraries split the K dimension (split-K) or run a persistent kernel that hands equal shares of the inner loop to each SM (stream-K). Small decode GEMMs hit the same tail effect.
A legacy reduction ends with an unrolled last-warp step: if (tid < 32) it adds sdata[tid + 32], then sdata[tid + 16] and so on through a volatile pointer, with no synchronization. It worked for years and now returns wrong sums intermittently. What changed, and how do you fix it?
HardExecution Model: Warps, Divergence and Occupancy
Sample Answer
The code assumes the 32 lanes of a warp execute in lockstep. Since Volta (compute capability 7.0), independent thread scheduling gives each thread its own program counter, and the hardware may interleave or reorder lanes between instructions, so lane 0 can read sdata[16] before lane 16 has written its partial. volatile stops the compiler from caching values in registers; it does not order threads. CUDA 13 toolkits only target compute capability 7.5 and newer, so every GPU you compile for has this behaviour. Fix it by putting __syncwarp() between steps, or better, by replacing the shared-memory steps with __shfl_down_sync, which needs no shared memory and names the participating lanes in its mask. The old mask-less intrinsics such as __shfl_down are not available for compute capability 7.x and later.
Most real kernels are limited by memory, not math, so memory questions take up a large share of any CUDA loop. Interviewers hand you an indexing expression or a profiler number and ask what the hardware fetches, where the data lives, and why the kernel is slow.
The idea that matters most: global memory moves in 32-byte sectors, and a warp-wide load costs the number of distinct sectors its 32 addresses touch. Shared memory has a similar rule with 32 banks of 4 bytes. Almost every fix in this section is a change of layout or index mapping that makes neighbouring lanes touch neighbouring words.
Memory Hierarchy, Coalescing and Bank Conflicts
Particles are stored as an array of struct { float x, y, z, mass; }. A hot kernel reads only x, one particle per thread, and runs at a quarter of the bandwidth you expected. Why, and what do you change?
EasyMemory Hierarchy, Coalescing and Bank Conflicts
Sample Answer
Lane i reads float 4 * i, a 16-byte stride. Global memory is served in 32-byte sectors, so a warp's 32 reads span 512 bytes and touch 16 sectors, of which it uses 128 bytes: 25% efficiency. A fully coalesced float load touches 4 sectors, and Nsight Compute reports this as sectors per request, 4 ideal against 16 here. Store the hot fields as a struct of arrays (all x values contiguous) and the same read becomes perfectly coalesced. The nuance: if the kernel uses all four fields, the array of structs is fine when each thread loads its particle as one aligned float4, a single 16-byte vector load per lane that touches the same 16 sectors and uses all of them.
A naive matrix transpose out[x * h + y] = in[y * w + x] runs at a fraction of copy bandwidth on large matrices. Explain why and write the standard fix.
MediumMemory Hierarchy, Coalescing and Bank Conflicts
Sample Answer
With threadIdx.x driving x, the reads are coalesced but the writes are not: neighbouring lanes write addresses h floats apart, so each warp store touches 32 sectors and uses 4 bytes of each, 12.5% efficiency. No index swap fixes this, because a transpose must read along rows and write along columns. Stage a 32 × 32 tile in shared memory: read it along rows (coalesced), barrier, then write rows of the output while reading the tile down its columns. A [32][32] tile would make those column reads a 32-way bank conflict, because each row is 32 words; padding to [32][33] makes the stride 33, which is odd, so the 32 lanes land in 32 different banks. This version uses 32 × 8 thread blocks, each thread handling four rows.
C++
1#defineTILE322__global__ voidtranspose(float* out,constfloat* in,int w,int h){// block 32 x 83 __shared__ float tile[TILE][TILE +1];// +1: no bank conflicts4int x = blockIdx.x * TILE + threadIdx.x, y = blockIdx.y * TILE + threadIdx.y;5for(int j =0; j < TILE; j += blockDim.y)6if(x < w && y + j < h) tile[threadIdx.y + j][threadIdx.x]= in[(y + j)* w + x];7__syncthreads();8 x = blockIdx.y * TILE + threadIdx.x; y = blockIdx.x * TILE + threadIdx.y;9for(int j =0; j < TILE; j += blockDim.y)10if(x < h && y + j < w) out[(y + j)* h + x]= tile[threadIdx.x][threadIdx.y + j];11}
Nsight Compute flags shared-memory bank conflicts in a kernel where thread t reads s[4 * t] from a __shared__ float array. How many ways is the conflict, and what are your options?
MediumMemory Hierarchy, Coalescing and Bank Conflicts
Sample Answer
Shared memory has 32 banks, each 4 bytes wide, and float k lives in bank k % 32. With a stride of 4, lanes 0, 8, 16 and 24 all hit bank 0 (words 0, 32, 64, 96), and every other bank gets the same treatment, so it is a 4-way conflict: the access is replayed in 4 passes (the profiler counts 4 wavefronts instead of 1). Lanes reading the same word do not conflict, because the value is broadcast. Any odd stride is conflict-free, since an odd number shares no factor with 32. Options: change the mapping so consecutive lanes read consecutive words, pad each row by one word, or swizzle the index with an XOR, which libraries such as CUTLASS use because it keeps tiles dense.
The compiler output for your kernel shows Used 255 registers and several hundred bytes of spill stores and spill loads, and the kernel got slower after you added a per-thread float acc[64] array. What is going on?
MediumMemory Hierarchy, Coalescing and Bank Conflicts
Sample Answer
A thread can use at most 255 registers. Values that do not fit are spilled to local memory, which is per-thread storage in device memory (off-chip, cached in L1 and L2), so each spill becomes a load or store with global-memory latency. Per-thread arrays are a common cause: if the array is indexed with a value the compiler cannot resolve at compile time, it lives in local memory even without register pressure. Fixes: make the indexing static (fully unroll loops with constant trip counts), shrink the per-thread tile, move the data to shared memory, or remove a register cap set by -maxrregcount or __launch_bounds__. See it with nvcc --ptxas-options=-v, and in Nsight Compute as local memory traffic and LG Throttle stalls.
A colleague moved a 16 KB lookup table into __constant__ memory to make it fast. Each thread indexes the table with its own data value, and the kernel got slower. Why, and where should the table live?
MediumMemory Hierarchy, Coalescing and Bank Conflicts
Sample Answer
Constant memory is cached and is fastest when every lane of a warp reads the same address: the value is broadcast, about as cheap as a register. When lanes read different addresses, the Best Practices Guide says the accesses are serialized, so a data-dependent lookup can take up to 32 times as long per warp. Constant memory suits kernel-wide parameters and coefficients that all threads read in step, such as a convolution filter tap. Its total size is 64 KB. For a table indexed per thread, use ordinary global memory read through a const __restrict__ pointer, which lets the compiler use the read-only load path, or copy the table into shared memory at block start: 16 KB fits easily and gives one-pass access unless lanes collide on banks.
Your attention kernel needs 100 KB of shared memory per block on an H100. The launch fails, even though the H100 advertises far more shared memory per SM. What is missing, and how does the same kernel fare on an A100 and an RTX card?
HardMemory Hierarchy, Coalescing and Bank Conflicts
Sample Answer
Static __shared__ arrays are capped at 48 KB per block. Above that you must use dynamic shared memory (extern __shared__) and opt in per kernel with cudaFuncSetAttribute and cudaFuncAttributeMaxDynamicSharedMemorySize, then pass the size as the third launch parameter. The per-block maximum depends on the architecture: 227 KB on compute capability 9.0 (H100), 163 KB on 8.0 (A100), 99 KB on 8.6, 8.9 and 12.0 consumer parts. Per SM it is 228, 164 and 100 KB, carved out of the same array as L1, and each block also reserves 1 KB. So two 100 KB blocks fit on an H100 SM, one on an A100, and the kernel cannot launch on an RTX card. Query cudaDevAttrMaxSharedMemoryPerBlockOptin instead of hard-coding.
C++
1__global__ voidattn(constfloat* q,float* o){2extern __shared__ float smem[];// size comes from the launch3// ... use smem ...4}56voidlaunch_attn(constfloat* q,float* o, dim3 grid, dim3 block, cudaStream_t s){7int bytes =100*1024;8cudaFuncSetAttribute(attn, cudaFuncAttributeMaxDynamicSharedMemorySize, bytes);9 attn<<<grid, block, bytes, s>>>(q, o);10}
Synchronization, Atomics and Warp-Level Primitives
Synchronization bugs are the CUDA bugs that pass tests and fail in production: a missing barrier, a barrier inside a branch, an atomic on a hot address, a shuffle mask that names lanes that are not there. Interviewers use them to check whether you know exactly what each primitive guarantees and what it costs.
The idea that matters most: CUDA gives you cheap synchronization at three scopes. A warp synchronizes through _sync intrinsics and __syncwarp(), a block through __syncthreads(), and a grid only through a kernel boundary or a cooperative launch. Pick the narrowest scope that makes the data dependency safe.
Synchronization, Atomics and Warp-Level Primitives
A block reduction loop does if (tid < s) { buf[tid] += buf[tid + s]; __syncthreads(); }. It hangs on some GPUs and gives wrong sums on others. Explain and fix it.
EasySynchronization, Atomics and Warp-Level Primitives
Sample Answer
__syncthreads() may appear in conditional code only if the condition evaluates the same way for every thread in the block. Here threads with tid >= s skip the barrier, so the behaviour is undefined: it can hang, or threads can read partial sums before they are written. Move the barrier out of the branch so every thread reaches it on every iteration, and keep only the work inside the condition. A related detail: the barrier waits for all non-exited threads, so an early return for out-of-range threads does not deadlock it, but those threads never write their shared slot, so load the identity value (0 for a sum) instead of returning. compute-sanitizer --tool synccheck reports this class of bug.
C++
1for(int s = blockDim.x /2; s >0; s >>=1){2if(tid < s) buf[tid]+= buf[tid + s];3__syncthreads();// every thread, every iteration4}
A 256-bin histogram kernel calls atomicAdd on global memory for every input byte. On real images, where most pixels are dark, it is 10 times slower than on random data. Why, and how do you fix it?
MediumSynchronization, Atomics and Warp-Level Primitives
Sample Answer
Global atomics are performed in the L2 cache, and updates to the same address are applied one at a time. On random data the updates spread over 256 addresses; on skewed data most of them queue on a few bins, and the busiest bin sets the kernel time. Privatize: each block counts into its own copy of the histogram in shared memory, where atomics run inside the SM, using a grid-stride loop so each block covers many elements. Then merge with one global atomic per bin per block. 256 unsigned int bins are 1 KB of shared memory, so this fits easily; 65,536 bins would be 256 KB, more than an A100 or H100 SM has.
C++
1__global__ voidhist256(constunsignedchar* in, size_t n,unsignedint* bins){2 __shared__ unsignedint local[256];3for(int b = threadIdx.x; b <256; b += blockDim.x) local[b]=0;4__syncthreads();5for(size_t i = blockIdx.x *(size_t)blockDim.x + threadIdx.x; i < n;6 i +=(size_t)gridDim.x * blockDim.x)7atomicAdd(&local[in[i]],1u);8__syncthreads();9for(int b = threadIdx.x; b <256; b += blockDim.x)atomicAdd(&bins[b], local[b]);10}
Write a block-wide sum using warp shuffles. How many __syncthreads() calls does it need compared with a shared-memory tree for a 256-thread block?
MediumSynchronization, Atomics and Warp-Level Primitives
Sample Answer
__shfl_down_sync(mask, v, offset) returns v from the lane offset places higher, so five steps with offsets 16, 8, 4, 2, 1 leave the warp total in lane 0 with no shared memory. A block sum reduces each warp, writes one partial per warp to a small shared array, waits at one barrier, and lets warp 0 reduce the partials. For 256 threads a shared-memory tree needs 9 barriers (one after the load and one per halving step); this needs 1. Two traps: the full mask 0xffffffff is only valid if all 32 lanes are active and reach the call, so blocks must be a multiple of 32 with out-of-range threads contributing 0; and calling block_sum twice in a row needs a barrier before partial is reused.
C++
1__device__ floatwarp_sum(float v){2for(int off =16; off >0; off >>=1)3 v +=__shfl_down_sync(0xffffffffu, v, off);4return v;// lane 0 holds the warp total5}6__device__ floatblock_sum(float v){// blockDim.x: multiple of 32, <= 10247 __shared__ float partial[32];8int lane = threadIdx.x %32, warp = threadIdx.x /32;9 v =warp_sum(v);10if(lane ==0) partial[warp]= v;11__syncthreads();12 v =(threadIdx.x < blockDim.x /32)? partial[lane]:0.0f;13if(warp ==0) v =warp_sum(v);14return v;// valid in thread 015}
Your loss reduction gives results that differ in the last few digits between identical runs. The kernel reduces within each block and then does one atomicAdd per block into the output. Is this a bug, and how would you make it deterministic?
MediumSynchronization, Atomics and Warp-Level Primitives
Sample Answer
It is expected behaviour, not memory corruption. Floating-point addition is not associative, and the order in which blocks reach the atomic depends on scheduling, which changes from run to run, so the rounding changes too. Integer atomics would give the same total every time; float atomics do not. To get bit-identical results, have each block write its partial sum to an array at index blockIdx.x, then reduce that array in a fixed order with a second small kernel (or a fixed-order tree in the last block). The cost is one extra launch of a few microseconds and a small buffer. Also expect GPU sums to differ slightly from a sequential CPU loop, because a tree adds in a different order. Compare with a tolerance, not equality.
An iterative Jacobi solver needs every block to finish iteration k before any block starts iteration k + 1. A teammate wrote a global-memory spin barrier with an atomic counter, and the program sometimes hangs. What are the correct options?
HardSynchronization, Atomics and Warp-Level Primitives
Sample Answer
A spin barrier assumes every block is resident at once. With an ordinary <<<>>> launch that is not guaranteed: if the grid is larger than what fits, resident blocks spin forever waiting for blocks that cannot be scheduled until they exit. Option one is a kernel launch per iteration, since consecutive kernels in one stream are ordered; the cost is a few microseconds per launch, which a CUDA graph can cut. Option two is a persistent kernel using cooperative groups: cg::this_grid().sync(), launched with cudaLaunchCooperativeKernel. That API only succeeds if the whole grid can be co-resident, so size the grid from cudaOccupancyMaxActiveBlocksPerMultiprocessor times the SM count and use a grid-stride loop. As of CUDA 13 cooperative groups no longer support multi-device synchronization.
C++
1#include<cooperative_groups.h>2namespace cg = cooperative_groups;34__global__ voidjacobi(float* a,float* b,unsignedlonglong n,int iters){5 cg::grid_group grid = cg::this_grid();6for(int it =0; it < iters;++it){7for(unsignedlonglong i = grid.thread_rank(); i < n; i += grid.num_threads())8if(i >0&& i +1< n) b[i]=0.5f*(a[i -1]+ a[i +1]);9 grid.sync();// all blocks, every iteration10float* t = a; a = b; b = t;11}12}13// Host: void* args[] = {&d_a, &d_b, &n, &iters};14// cudaLaunchCooperativeKernel((void*)jacobi, blocks, threads, args, 0, stream);
A kernel computes if (i < n) { float v = x[i]; v = warp_sum(v); ... } where warp_sum uses __shfl_down_sync(0xffffffff, ...). It works when n is a multiple of 32 and returns garbage, or hangs, otherwise. Explain, and give two fixes.
HardSynchronization, Atomics and Warp-Level Primitives
Sample Answer
The mask in a _sync intrinsic names the lanes that will execute it, and those lanes must all reach it. In the last warp, lanes with i >= n skip the branch, yet the mask names them, which is undefined behaviour. Even with a correct partial mask, __shfl_down_sync reading from an inactive lane returns an undefined value, so the tree would add garbage. Fix one: keep every lane in the shuffle and give out-of-range lanes the identity, float v = (i < n) ? x[i] : 0.0f;, then reduce outside the branch. Fix two: use cooperative groups, for example cg::coalesced_threads() with cg::reduce(g, v, cg::plus<float>()), which handles the active set for you. Do not build the mask from __activemask(); the guide describes it as a snapshot that guarantees nothing about convergence.
Performance rounds test whether you can say how fast a kernel should be before you measure it, then measure it honestly. Expect a roofline estimate, a profiler report to interpret, a timing bug to spot, and a host-side pipeline that fails to overlap.
The idea that matters most: every kernel has a floor, the larger of FLOPs divided by peak math and bytes divided by peak bandwidth. Compare the measured time with that floor first. If the gap is small, stop optimizing the kernel; if the GPU is idle between kernels, the problem is on the host, and streams, pinned memory or CUDA graphs are the fix.
Performance Analysis, Streams and Concurrency
A vector add of 100 million floats takes 700 µs on an A100 80GB. Is it memory-bound or compute-bound, and is there anything left to optimize inside the kernel?
EasyPerformance Analysis, Streams and Concurrency
Sample Answer
It reads two floats and writes one per element: 12 bytes for 1 FLOP, an arithmetic intensity of about 0.083 FLOP/byte. The A100's FP32 ridge point is 19.5 TFLOPS ÷ about 2.0 TB/s, roughly 9.75 FLOP/byte, so the kernel sits far left of the ridge and is memory-bound. Its floors: 1.2 GB ÷ 2.0 TB/s is 600 µs for memory, against about 5 µs for math. At 700 µs it already achieves about 86% of peak bandwidth (effective bandwidth is bytes read plus bytes written over time). Faster math changes nothing. The only real gains come from moving fewer bytes: fuse it with the kernels before and after so the intermediate never touches HBM, or store the data in BF16 or FP16 to halve the traffic.
Nsight Compute reports a kernel at 18% Compute (SM) Throughput and 22% Memory Throughput. The kernel is slow. How do you read this, and what do you check next?
MediumPerformance Analysis, Streams and Concurrency
Sample Answer
Both percentages low means the kernel is latency-bound: neither the math units nor any memory unit is busy, so the hardware is mostly waiting. First rule out host-side gaps in Nsight Systems, then read the report top-down. Launch Statistics: is the grid smaller than the number of SMs, or a fraction of one wave? Occupancy: what achieved occupancy and how many eligible warps per scheduler? Warp State Statistics: if Long Scoreboard dominates while DRAM throughput is low, there are not enough bytes in flight, so add warps or independent loads per thread (for example float4 loads or several elements per thread). If Barrier dominates, work is uneven between warps. Collect it with ncu --set full -k regex:name -s 5 -c 1 -o report ./app, skipping warm-up launches.
A colleague times a kernel with std::chrono around the launch and gets 8 µs whatever the input size. Then they add a cudaDeviceSynchronize() and the first measurement is far slower than the rest. Explain both effects and show how you would time it.
EasyPerformance Analysis, Streams and Concurrency
Sample Answer
Kernel launches are asynchronous: the host call returns once the launch is queued, so the 8 µs is CPU launch cost, not kernel time. Adding a synchronize fixes that, but the first launch also pays one-time costs such as lazy module loading and cold caches. Time on the GPU with events recorded in the same stream, after a warm-up launch, over several repetitions, and wait on the stop event before reading it. cudaEventElapsedTime returns milliseconds. Check cudaGetLastError() after the launches too: a launch that failed with a bad configuration runs in zero time and looks very fast.
C++
1cudaEvent_t start, stop;2cudaEventCreate(&start);3cudaEventCreate(&stop);4kernel<<<grid, block,0, s>>>(d_in, d_out, n);// warm-up5cudaEventRecord(start, s);6for(int r =0; r <20;++r) kernel<<<grid, block,0, s>>>(d_in, d_out, n);7cudaEventRecord(stop, s);8cudaEventSynchronize(stop);9float ms =0.0f;10cudaEventElapsedTime(&ms, start, stop);// milliseconds11printf("%.3f ms per launch, %s\n", ms /20,cudaGetErrorString(cudaGetLastError()));
A program copies 200 MB to the GPU, runs a 6 ms kernel and copies the result back. The copies take 8 ms each way, so a run takes 22 ms. How would you overlap the work, and what is the best you can hope for?
MediumPerformance Analysis, Streams and Concurrency
Sample Answer
Split the data into chunks and give each chunk its own stream: copy in, kernel, copy out. Work in one stream is ordered; work in different streams can run concurrently on separate hardware, the copy engines and the SMs. Data center GPUs have at least two copy engines and PCIe is full duplex (about 25 GB/s each way on PCIe 4.0 x16), so an upload, a download and a kernel can all be in flight. With four chunks the run drops from 22 ms to about 11.5 ms. The floor is the slowest stage, 8 ms here. The host buffer must be pinned (cudaMallocHost): with pageable memory cudaMemcpyAsync reverts to synchronous behaviour and does not overlap. Allocate device memory once, outside the loop.
C++
1float*h,*d;2cudaMallocHost((void**)&h, N *sizeof(float));// pinned: required for overlap3cudaMalloc(&d, N *sizeof(float));// allocate once, before the loop4cudaStream_t s[4];5for(int k =0; k <4;++k)cudaStreamCreateWithFlags(&s[k], cudaStreamNonBlocking);6size_t chunk =(N +3)/4;7for(int k =0; k <4;++k){8 size_t off = k * chunk;9if(off >= N)break;// tiny N: fewer chunks10 size_t len = std::min(chunk, N - off);11cudaMemcpyAsync(d + off, h + off, len *sizeof(float), cudaMemcpyHostToDevice, s[k]);12 process<<<(len +255)/256,256,0, s[k]>>>(d + off, len);13cudaMemcpyAsync(h + off, d + off, len *sizeof(float), cudaMemcpyDeviceToHost, s[k]);14}15cudaDeviceSynchronize();
Your multi-stream pipeline, built with streams from cudaStreamCreate, overlapped nicely until someone added a debugging cudaMemset and a stream-less helper kernel launch. Now Nsight Systems shows everything serialized. Why?
MediumPerformance Analysis, Streams and Concurrency
Sample Answer
Calls without a stream go to the legacy default stream, stream 0. It has a special rule: it synchronizes with all other blocking streams, so work in it waits for everything already queued in those streams, and later work in them waits for it. One stray call in the loop serializes the whole pipeline. Streams from cudaStreamCreate are blocking by default. Fixes: pass your stream to every call (cudaMemsetAsync(ptr, 0, bytes, s), <<<g, b, 0, s>>>), create streams with cudaStreamNonBlocking, or compile with nvcc --default-stream per-thread so each host thread gets an ordinary default stream. Watch for other implicit sync points too: device memory allocation, pinned host allocation and cudaFree between chunks, and host reads of device values such as .item() in PyTorch.
An LLM decode step at batch 1 measures 15 ms when the roofline says the GPU work should take under 10 ms. Nsight Systems shows short kernels separated by gaps and a busy CPU launch thread. Explain, and show how CUDA graphs help.
HardPerformance Analysis, Streams and Concurrency
Sample Answer
Launches are asynchronous, but each still costs CPU time, and short kernels drain the queue faster than the CPU refills it. A step takes the larger of GPU time and launches × CPU cost per launch. An unfused Llama 3.1 8B decode step is about 548 kernels (17 per layer × 32, plus 4): at 10 µs each that is 5.5 ms of CPU, and at 20 µs it is 11 ms, longer than the GPU work. A CUDA graph is captured and instantiated once, then replayed with one cudaGraphLaunch per step. Capture needs fixed shapes and addresses and no allocation or host sync inside, so engines capture several batch sizes and pad. Fusion cuts kernel count too; graphs do not make a GPU-bound step faster.
C++
1cudaGraph_t graph;2cudaGraphExec_t exec;3cudaStreamBeginCapture(s, cudaStreamCaptureModeGlobal);4for(int layer =0; layer <32;++layer)run_layer(layer, s);// recorded, not run5cudaStreamEndCapture(s,&graph);6cudaGraphInstantiate(&exec, graph,0);// once7for(int step =0; step < max_new_tokens;++step)8cudaGraphLaunch(exec, s);// one CPU call per step9cudaStreamSynchronize(s);
ML infrastructure and inference roles ask you to write or explain the kernels that dominate model time: matrix multiply, softmax, attention and quantized matrix-vector products. They also want to know when you would write CUDA C++ at all, as opposed to Triton or a library, and how you debug a kernel that gives wrong answers.
The idea that matters most: deep learning kernels are fast when data is reused on chip and intermediate results never round-trip through HBM. Tiling, fusion, FlashAttention and fused dequantization are all the same move: raise arithmetic intensity by keeping bytes in registers and shared memory, then use Tensor Cores for whatever math is left.
Kernels in Practice and the ML Angle
Write a shared-memory tiled matrix multiply for square FP32 matrices of any size, and explain what tiling buys over the naive kernel.
MediumKernels in Practice and the ML Angle
Sample Answer
The naive kernel gives each thread one output and reads a full row of A and column of B from global memory: two 4-byte loads per multiply-add, about 0.25 FLOP per requested byte. Tiling has a block of TILE × TILE threads load one tile of A and one of B into shared memory, then each thread does TILE multiply-adds from shared memory, so each global load serves TILE operations: intensity TILE ÷ 4, so 4 FLOP/byte at TILE 16. Two barriers per phase: one after the loads so tiles are complete, one after the compute so no warp overwrites a tile another warp is still reading. Out-of-range threads load zeros but still reach both barriers. Next come register tiling and Tensor Cores; production code calls cuBLAS or CUTLASS.
C++
1#defineT162__global__ voidmatmul(constfloat* A,constfloat* B,float* C,int N){3 __shared__ float As[T][T], Bs[T][T];4int r = blockIdx.y * T + threadIdx.y, c = blockIdx.x * T + threadIdx.x;5float acc =0.0f;6for(int k0 =0; k0 < N; k0 += T){7 As[threadIdx.y][threadIdx.x]=(r < N && k0 + threadIdx.x < N)? A[r * N + k0 + threadIdx.x]:0.0f;8 Bs[threadIdx.y][threadIdx.x]=(c < N && k0 + threadIdx.y < N)? B[(k0 + threadIdx.y)* N + c]:0.0f;9__syncthreads();10for(int k =0; k < T;++k) acc += As[threadIdx.y][k]* Bs[k][threadIdx.x];11__syncthreads();12}13if(r < N && c < N) C[r * N + c]= acc;14}
Row-wise softmax over a [batch, 4096] FP32 tensor runs as separate max, subtract, exp, sum and divide kernels. Some rows produce NaN. Design a better kernel.
MediumKernels in Practice and the ML Angle
Sample Answer
NaN: expf overflows to infinity above about 88.7 in FP32, and infinity over infinity is NaN. Subtract the row max first; softmax is unchanged by the shift and every exponent is at most 0. Speed: softmax is memory-bound, and five kernels mean several full passes over HBM. Fuse it: one block per row (or one warp for short rows), each thread holds 4 of the 4,096 values in registers, a block reduction finds the max, another finds the sum of exp(x - max), then the row is written once. That is one read and one write, against eight passes for the unfused chain. For rows too long to hold on chip, online softmax keeps a running max and sum, rescaling the sum when the max grows.
An FP32 training job on an A100 shows its matmuls at about 19 TFLOPS. How do you get more out of the GPU, and what can break numerically?
MediumKernels in Practice and the ML Angle
Sample Answer
Those matmuls run on the FP32 CUDA cores, whose A100 peak is 19.5 TFLOPS. Tensor Cores do dense TF32 at 156 TFLOPS and FP16 or BF16 at 312 (about 990 BF16 and 1,979 FP8 on an H100 SXM). In PyTorch, TF32 for FP32 matmuls is off by default; enable it with torch.set_float32_matmul_precision("high"), or train in BF16 autocast. Numerics: Tensor Cores take FP16 or BF16 inputs and accumulate in FP32, which is what keeps long dot products accurate. FP16 tops out at 65,504 and underflows small gradients, so it needs dynamic loss scaling; BF16 has FP32's exponent range but only 8 significant bits, so keep FP32 master weights and optimizer states. Dimensions that are multiples of 8 for FP16 use the Tensor Core paths most efficiently.
Explain why FlashAttention is faster than standard attention for sequence length 4,096, head dimension 64 and 32 heads in FP16 on an A100. Does it reduce FLOPs?
HardKernels in Practice and the ML Angle
Sample Answer
Standard attention writes the N × N scores and probabilities to HBM and reads them back: about 4Nd + 4N² elements per head, roughly 4.4 GB here, at least 2.2 ms at 2.0 TB/s, while the matmuls are about 137 GFLOP, 0.44 ms at 312 TFLOPS. FlashAttention tiles Q, K and V so each block of scores lives only in registers and shared memory. Online softmax keeps the result exact, not approximate: per query row it tracks the running max m, sum l and unnormalized output acc, rescales l and acc by exp(m_old - m_new) when the max grows, and divides at the end. Traffic falls to about 1.1 GB, near the math time. FLOPs do not drop; the backward pass even recomputes score tiles from a stored logsumexp.
Your team needs a custom fused kernel for a new attention variant. When would you write it in Triton, and when in CUDA C++?
MediumKernels in Practice and the ML Angle
Sample Answer
Triton lets you write block-level programs in Python: tl.program_id replaces blockIdx, masks replace bounds checks, and the compiler maps elements to threads, coalesces contiguous offsets, manages shared memory and barriers, and emits Tensor Core instructions for tl.dot. It is the fastest route to a fused kernel that is close to hand-tuned speed, and it is what torch.compile generates for fused elementwise and reduction code. Choose CUDA C++ (often through CUTLASS) when you need warp-level control such as shuffles or warp specialization, direct use of Hopper and Blackwell features like TMA, wgmma or tcgen05, or deployment in a C++ server or TensorRT plugin without a Python JIT. Either way, you still choose layouts, block sizes and fusion boundaries, and you still check the result against the roofline.
A kernel passes unit tests but occasionally returns wrong values in the last block, and results differ between runs. Which tools do you reach for, and in what order?
EasyKernels in Practice and the ML Angle
Sample Answer
Start with error checking: every runtime call returns a cudaError_t, launch errors appear from cudaGetLastError(), and asynchronous faults surface at the next synchronizing call, so CUDA_LAUNCH_BLOCKING=1 helps point at the failing launch. Then run compute-sanitizer on a small, deterministic input. memcheck (the default) finds out-of-bounds and misaligned accesses, typically a missing bounds check on the last tile. racecheck finds shared-memory hazards such as a missing barrier before a tile is overwritten. initcheck finds reads of uninitialized global memory. synccheck finds invalid barrier use, such as __syncthreads() in a divergent branch. Compile with -lineinfo so reports name source lines. The tools slow kernels down a lot, so never run them on a full workload.
After quantizing Llama 3.1 8B to INT8 weight-only, batch-1 decode on an H100 got slower than BF16. Why, and what kernel fixes it?
HardKernels in Practice and the ML Angle
Sample Answer
Batch-1 decode is a chain of matrix-vector products. Each weight is read once for one multiply-add, about 1 FLOP per byte in BF16, far below the H100's Tensor Core ridge of about 295, so time is weight bytes over bandwidth. INT8 should halve those bytes, but an unfused pipeline runs a separate dequantize kernel that reads 1 byte and writes a 2-byte BF16 weight, which the GEMV then reads again: 5 bytes per weight instead of 2. In the course's model at 80% of H100 bandwidth that is 37.5 GB and 14.0 ms per step, against 15.0 GB and 5.6 ms in BF16. Fuse the dequantization into the GEMV: load INT8 weights, convert in registers, apply the per-row scale once. That moves 7.5 GB, about 2.8 ms.
Know the per-SM limits of at least one GPU from memory (an A100 has 64 warps, 32 blocks, 65,536 registers and up to 164 KB of shared memory per SM) and practise turning a launch configuration into resident blocks. Then be ready to argue why low occupancy can be fine. The Occupancy lesson walks through the calculation and the Little's law argument.
Count sectors and banks for any index
Given in[f(threadIdx.x)], you should be able to say how many 32-byte sectors a warp touches and, for shared memory, how many ways it conflicts. Work through Coalescing and Shared memory and bank conflicts until the padding trick is automatic.
Write the classic kernels without notes
A block reduction with warp shuffles, a privatized histogram, a tiled matmul and a fused softmax cover most live-coding prompts. Write each one, then explain where every barrier goes and why. The Reduction and Kernel round lessons build them step by step.
Open every performance answer with a floor
Before discussing fixes, compute FLOPs over peak and bytes over bandwidth for the GPU in question, then compare with the measured time. Interviewers notice when a candidate knows a kernel is already at 86% of bandwidth. The roofline model and Profiling with Nsight Compute show how to get both numbers and read a report in order.
Connect kernels to model-level costs
For ML and inference roles, practise explaining FlashAttention with byte counts, why batch-1 decode is memory-bound, and when launch overhead dominates a step. The FlashAttention lesson and Fusion, CUDA graphs and launch overhead cover these with real numbers.
How Ready Are You for CUDA Interviews?
1 / 6
Execution Model
A kernel branches on threadIdx.x % 2, with 20 instructions on each path. What does each warp pay?
Frequently Asked Questions
Is CUDA asked in machine learning engineer interviews?
It depends on the team. Research and product ML roles rarely test CUDA directly, though they may ask why a model is memory-bound or how mixed precision works. ML infrastructure, inference, compiler and GPU kernel roles do test it, often with a live kernel (a reduction, softmax or tiled matmul) and a performance discussion built on the roofline model and profiler output. If the job description mentions kernels, Triton, TensorRT or GPU performance, prepare for CUDA questions.
Which CUDA version should I know for interviews in 2026?
Interviews test concepts that have been stable for years: warps, the memory hierarchy, barriers, streams. Still, know the current context. CUDA 13 toolkits dropped offline compilation for GPUs older than compute capability 7.5, so every target has independent thread scheduling and you should use the _sync warp intrinsics. Be able to name what Hopper and Blackwell added (thread block clusters, TMA, FP8, FP4 Tensor Cores) at a high level, even if you mostly work through libraries.
How long does it take to prepare for a CUDA interview?
If you already write C++ and know basic GPU concepts, three to four weeks of steady practice is usually enough: one week on the execution model and memory, one on writing the classic kernels, one on profiling and streams, and time for ML kernels if the role needs them. Starting from scratch, plan on two to three months. Writing kernels and checking them against a reference teaches far more than reading.
Do I need access to a GPU to prepare?
It helps but is not required. Much of interview reasoning is arithmetic: sectors per request, bank conflicts, occupancy, roofline floors. You can practise that on paper or in a simulator. For timing and profiling, a cloud GPU instance for a few hours is enough to run Nsight Compute on your own kernels. Know the commands and how to read the report even if you profile only a handful of times.
Should I learn Triton or CUDA C++ first?
For ML roles, learn the CUDA execution and memory model first, because Triton answers still rely on it: coalescing, occupancy and the roofline do not go away. Then write kernels in whichever language the team uses. Triton gets you to fused kernels quickly and is what torch.compile emits. CUDA C++ remains the language of interview whiteboards, CUTLASS and inference engines, so be able to read and write a simple kernel in it either way.
What level of CUDA knowledge do inference engineer interviews expect?
Expect to explain why decode is memory-bound and prefill is compute-bound, what FlashAttention, paged attention and CUDA graphs fix, and how quantization changes the bytes moved per token. You should be able to read an Nsight Systems timeline and an Nsight Compute report, write a reduction or GEMV kernel, and estimate step time from model size and bandwidth. Deep Tensor Core programming is usually left to specialist kernel roles.
Written by
Dan Lee
Data & AI Lead
Dan is a seasoned data scientist and ML coach with 10+ years of experience at Google, PayPal, and startups. He has helped candidates land top-paying roles and offers personalized guidance to accelerate your data career.