The GPU Memory Hierarchy: Registers to VRAM

Topic Overview

Five Levels from Registers to VRAM

GPUs expose a multi-level memory hierarchy that programmers often manage explicitly. Understanding each level is essential for writing efficient compute kernels and shaders.

1. Registers: per-thread private storage, ~1 cycle latency
2. Shared memory / L1: on-chip scratchpad shared within a thread block, ~20 cycles
3. L2 cache: shared across all SMs, ~50-100 cycles
4. Global memory (VRAM): off-chip DRAM, ~400-800 cycles
5. Constant / texture caches: read-only paths optimized for broadcast and 2D locality

If a kernel uses too many registers, the compiler spills them to local memory backed by VRAM. That spill is orders of magnitude slower than keeping values in registers. This tradeoff is called register pressure.

Latency Hiding, Not Just Bandwidth

GPUs have enormous bandwidth but terrible latency. The scheduler hides latency by switching to other ready warps while one warp waits on a global load. For example, a kernel that performs one global read per thread with low occupancy may stall frequently because too few warps are available to cover the ~400 cycle VRAM round trip.

cCopy
Loading…

Shared memory requires explicit allocation and copy. L1 cache is automatic but smaller. The key insight: maximize active warps and minimize global traffic through shared memory staging and coalesced access.

For This Exercise

You will model latency and bandwidth at each hierarchy level and compute how many warps a kernel needs to hide global memory stalls. This task asks you to print per-level access costs and explain when register spills would hurt occupancy. The numbers you compute here connect directly to the coalescing and occupancy exercises that follow.

Your Task

Model the GPU memory hierarchy well enough to see why kernels are written the way they are:

• registers (1 cycle) and shared memory (20 cycles);
• an L1 and an L2 cache of 128-byte lines (30 and 200 cycles on a hit);
• DRAM (600 cycles).

Simulate individual global loads through fully-associative LRU caches, sweep over memory with different strides, and estimate how shared-memory tiling cuts the global traffic of a matrix multiply.

Input

# starts a comment.

cCopy
Loading…

Rules

• Line: addr / 128.
• Load:
  1. an L1 hit costs the L1 latency;
  2. otherwise an L2 hit costs the L2 latency, and the line is inserted into L1;
  3. otherwise it goes to DRAM (DRAM latency, one DRAM line), and the line is inserted into L2, then L1.
• LRU: a hit refreshes the line's recency in that cache. Inserting into a full cache evicts its least recently used line.
• Every access adds its latency to the running cycle total, with no overlap: this is a latency-serialized model.
• matmul N TILE (TILE must divide N, N ≤ 4096):
  • naive global loads = 2N³ (each output reads a row and a column);
  • tiled global loads = 2N³/TILE, plus 2N³ shared loads;
  • cycles are loads × latency, with shared loads at the shared latency;
  • the reduction factor is the naive/tiled ratio;
  • arithmetic intensity = 2N³ FLOPs / (loads × 4 bytes), truncated to 2 decimals.

Output

cCopy
Loading…

• The per-load average in sweep is truncated. The hit percentage is rounded half up to 1 decimal (0.0% with no loads).
• Use access for 1 access and accesses otherwise.
• Errors:
  • latency: LEVEL CYCLES ... (levels reg shared l1 l2 dram)
  • l1: LINES (1-64) (or l2)
  • load: ADDR, sweep: START COUNT STRIDE
  • reg: COUNT (or shared)
  • matmul: N TILE (TILE must divide N, N <= 4096)
  • unknown command X

Example

Input:

cCopy
Loading…

Output:

cCopy
Loading…

Requirements

1. Track the caches per line, not per byte. That is why neighbouring threads' loads are cheap and strided ones are not.
2. Implement LRU with a recency stamp per line, refreshed on every hit.

Hidden tests cover LRU eviction with tiny caches (L2 catching L1 misses), changed latencies, a sweep that thrashes the caches, a full-tile matmul, and argument errors.