What We're Really Explaining Here

A GPU is fundamentally a massively parallel processor optimized for doing the same operation on thousands of data elements simultaneously. Understanding how it works requires understanding why that parallelism exists, how the hardware is structured to support it, and how both graphics and general-purpose compute workloads map onto that structure.

Let's build up the mental model from first principles rather than jumping straight to buzzwords like "CUDA cores" or "shader pipelines."

Why GPUs Exist: The Motivation

A CPU is designed for latency-sensitive, sequential work. It has a small number of powerful cores (typically 8 to 64), large caches, branch predictors, and out-of-order execution hardware. All of that complexity exists to make a single thread of work run as fast as possible.

Graphics rendering is fundamentally different. To render a 1080p frame at 60fps, you need to compute color values for roughly 2 million pixels, 60 times per second. Each pixel calculation is largely independent of the others. You don't need any single pixel to finish fast. You need all of them to finish together, and the computation per pixel is relatively simple and predictable.

This is the classic distinction: CPUs optimize for latency, GPUs optimize for throughput. A GPU achieves high throughput by trading away the complexity that makes CPUs fast at sequential tasks. Instead of 16 beefy cores, you get thousands of simpler cores that share control logic and instruction decoders.

Core Architecture: Streaming Multiprocessors

The fundamental building block of a modern GPU (using NVIDIA's terminology for concreteness) is the Streaming Multiprocessor (SM), called a Compute Unit in AMD's architecture. A high-end GPU might have 80 to 160 of these.

Each SM contains:

CUDA cores (or shader processors): These are simple ALUs that execute arithmetic instructions. A single SM might have 128 of these.
Tensor cores / specialized units: For matrix multiply-accumulate operations, critical for ML workloads.
Warp schedulers: Hardware units that schedule groups of threads onto the execution units.
Shared memory / L1 cache: Fast, low-latency memory shared among all threads in the SM. Think of this like a manually managed scratchpad, roughly 32KB to 128KB.
Register file: A large bank of registers shared across all active threads. This is actually quite big (e.g., 256KB per SM) because the GPU needs to keep many threads "in flight" simultaneously.

The key insight is that control logic (instruction fetch, decode, scheduling) is shared across all cores in an SM. This is how you can have 10,000 "cores" without the die area and power budget exploding: you amortize the expensive control hardware across many execution units.

SIMT: Single Instruction, Multiple Threads

The execution model GPUs use is called SIMT (Single Instruction, Multiple Threads). This is similar to SIMD (Single Instruction, Multiple Data) on CPUs, but with an important difference in how it's exposed to the programmer.

Threads are grouped into warps (NVIDIA) or wavefronts (AMD), typically 32 or 64 threads. All threads in a warp execute the same instruction at the same time, just on different data. The hardware provides the illusion that each thread has its own program counter, but in reality the warp executes lockstep.

This creates the classic GPU pitfall: warp divergence. Consider this code:

if (threadIdx.x % 2 == 0) {
    doExpensiveWork();
} else {
    doCheapWork();
}

Half the threads take the if-branch, half take the else-branch. Because the warp must execute in lockstep, the GPU serializes this: it first executes doExpensiveWork() with the odd-indexed threads masked off (doing nothing), then executes doCheapWork() with even-indexed threads masked off. You've effectively halved your throughput. This is why GPU code works best when threads follow the same control flow path.

The Memory Hierarchy

Memory is often the real bottleneck in GPU workloads. The hierarchy looks like this, from fastest to slowest:

Registers: Per-thread, sub-nanosecond access. The GPU keeps thousands of registers live simultaneously to support context switching between warps.
Shared memory / L1 cache: On-chip, within an SM. Latency around 20-40 cycles. Shared memory is explicitly managed by the programmer; L1 is automatic.
L2 cache: Shared across all SMs. Latency around 200 cycles.
VRAM (HBM or GDDR): The main GPU memory. High bandwidth (e.g., 1-3 TB/s for HBM3) but high latency, around 500-800 cycles.
System RAM: Accessed over PCIe, extremely slow from the GPU's perspective.

The reason GPUs can tolerate high VRAM latency is latency hiding through multithreading. When a warp issues a memory load and stalls waiting for data, the warp scheduler immediately switches to another ready warp and runs that instead. If you have enough warps in flight (high "occupancy"), memory latency is almost completely hidden. This is why GPUs have such large register files: to keep many warps "alive" simultaneously without context-switch overhead.

The practical implication is that you want coalesced memory accesses: adjacent threads should access adjacent memory addresses so the hardware can serve the entire warp's memory request in a single transaction rather than 32 separate ones.

How Graphics Workloads Map to This Hardware

The traditional graphics pipeline has several stages, and it's useful to understand how each maps to GPU hardware:

Vertex shading: Each vertex of a 3D mesh is transformed from model space to screen space. These are completely independent, so thousands of vertices can be processed in parallel.
Rasterization: Triangles are converted to fragments (candidate pixels). This is largely done in fixed-function hardware, not programmable shader cores.
Fragment/pixel shading: Each fragment gets its color computed (texture lookups, lighting calculations, etc.). Again, completely independent per fragment. This is the most shader-intensive stage.
Render output / depth test: Fragments are written to the framebuffer after passing depth tests. Fixed-function hardware.

Modern GPU drivers and APIs (Vulkan, Direct3D 12) manage how these pipeline stages are scheduled onto the SM cores. The driver compiles HLSL or GLSL shaders into GPU machine code, and the hardware dispatches vertex shader workloads across SMs in parallel, then fragment shader workloads, and so on.

General-Purpose Compute (GPGPU)

Once it became clear that "do the same thing to lots of data" is useful beyond graphics, the GPGPU model emerged. NVIDIA's CUDA and the open standard OpenCL expose the GPU hardware directly to compute workloads.

In CUDA's programming model:

You write a kernel: a function that will run on the GPU.
You launch that kernel with a grid of thread blocks. Each thread block maps to one SM, and threads within a block share that SM's shared memory and can synchronize with each other.
Threads across different blocks cannot easily communicate (they'd have to go through VRAM), so algorithm design needs to account for this.

Matrix multiplication is the canonical example of a well-optimized GPU workload. A naive implementation just has each thread compute one output element. An optimized implementation (like what cuBLAS does) tiles the matrices into blocks that fit in shared memory, loads tiles cooperatively, and reuses them, dramatically reducing VRAM bandwidth pressure. This is why GPU memory access patterns matter so much in practice.

Tensor Cores and the ML Angle

Modern GPUs include specialized Tensor Cores (NVIDIA A100/H100) or Matrix Cores (AMD CDNA). These perform a fused multiply-accumulate on small matrices (e.g., $4 \times 4$ or $16 \times 16$ tiles) in a single instruction. The operation is:

$$D = A \times B + C$$

where A, B, C, D are matrix tiles. This maps directly onto the inner loop of neural network training and inference (weight matrices times activation matrices). An H100 can do roughly 2,000 TFLOPS of FP8 tensor core operations, versus maybe 60 TFLOPS of standard FP32 on the same chip. The specialization pays off massively for this specific pattern.

The Real Constraint: Arithmetic Intensity

A useful way to think about whether a workload will be GPU-efficient is arithmetic intensity: the ratio of compute operations to memory bytes accessed.

$$\text{Arithmetic Intensity} = \frac{\text{FLOPs}}{\text{Bytes from memory}}$$

A GPU has a peak compute rate and a peak memory bandwidth. The ratio of these gives you the "ridge point" of the roofline model. Workloads above this intensity are compute-bound; workloads below it are memory-bandwidth-bound.

For example, large matrix multiplies have high arithmetic intensity (each element of the matrices is reused many times), so they're compute-bound and hit near-peak FLOP rates. Element-wise operations like ReLU have arithmetic intensity of roughly 1 FLOP per byte, making them heavily memory-bandwidth-bound. This is why kernel fusion (combining multiple element-wise ops into a single kernel pass) is such a common optimization in ML frameworks.

What GPUs Are Bad At

It's worth being honest about the limitations, because a good answer isn't just cheerleading for GPUs:

Divergent control flow: As discussed, branches that vary per thread kill efficiency. Recursive algorithms, decision trees, and similar workloads don't map well.
Irregular memory access: Pointer chasing (linked lists, graph traversal) generates non-coalesced memory accesses and thrashes the cache hierarchy.
Low parallelism tasks: If you only have 100 work items, you can't fill thousands of cores. The overhead of launching a GPU kernel (microseconds) exceeds the benefit.
PCIe transfer costs: Moving data between CPU and GPU memory over PCIe takes time. Workloads that constantly shuttle data back and forth will be dominated by transfer latency, not compute.

The general rule is: GPUs shine when you have massive, regular, independent parallelism with high arithmetic intensity. When those conditions aren't met, a CPU will often outperform or match a GPU while being much easier to program.

Putting It All Together

The through-line across all of this is a set of consistent design choices: favor throughput over latency, amortize control logic across many execution units, hide memory latency through massive multithreading rather than large caches, and provide specialized hardware (rasterizers, tensor cores) for the most common operations in the target workloads.

Understanding these principles lets you reason from first principles about why a particular workload will or won't benefit from GPU acceleration, and how to structure that workload (memory access patterns, thread block sizing, shared memory usage) to get the best performance when it does.

2794. GPU Operation Basics

Similar Questions