Throughout Parts III and IV we’ve repeated a phrase: “bandwidth-bound, not compute-bound.” It’s the central observation that drives FlashAttention, MoE inference, KV cache optimisation, and quantization. This section makes it rigorous. The “roofline model” (Williams 2009) classifies every operation by its arithmetic intensity — FLOPs per byte loaded from memory — and predicts whether the operation will be limited by peak FLOPs or by memory bandwidth. The boundary is the “ridge point”, set by hardware as peak_FLOPs / peak_bandwidth. For H100 at ~300 FLOPs/byte, almost every LLM operation EXCEPT large GEMM falls BELOW the ridge — meaning the hardware spends most of its time waiting for HBM. This is why LLM inference is bandwidth-bound; this is the structural fact that shapes the entire systems stack.

The GPU memory hierarchy

Modern GPUs have a 3-tier (sometimes 4-tier) memory hierarchy:

HBM GPU memory tier The off-chip stacked DRAM that holds GPU model weights and large activations. H100: 80 GB at 3.35 TB/s. B200: 192 GB at 8 TB/s. The 'slow tier' relative to on-chip SRAM (~5 TB/s per SM) but the 'fast tier' relative to system memory (~50 GB/s PCIe). All LLM weights live in HBM; inference cost is dominated by reading them from HBM into SRAM. is where everything lives. SRAM GPU memory tier The on-chip cache and shared memory on each Streaming Multiprocessor (SM) of a GPU. H100: 228 KB per SM, 132 SMs = ~30 MB total. Bandwidth: ~5 TB/s aggregate. Used by SM-local computations; CUDA shared memory and L1 cache come from this pool. 1000× lower latency than HBM. The 'fast working memory' that FlashAttention and other modern kernels target. is the fast scratchpad. The art of GPU kernel design is staging data through SRAM as much as possible.

Arithmetic intensity

The key metric: how many FLOPs of work does each loaded byte enable?

Arithmetic intensity performance metric The ratio of compute work (FLOPs) to memory traffic (bytes loaded), for a specific operation at a specific size. AI = FLOPs / bytes_loaded_from_HBM. An operation with high AI uses each byte many times for computation; an operation with low AI computes briefly with each byte. Hardware has a 'ridge point' AI = peak_FLOPs / peak_bandwidth; operations above this are compute-bound, below are memory-bound. For LLM ops on H100, AI typically ranges from 0.5 (dot product) to 1000+ (large GEMM). is the operation’s intrinsic property — independent of which GPU you run on. The hardware’s ridge point roofline concept The arithmetic intensity at which a hardware's peak compute and peak bandwidth balance: ridge = peak_FLOPs / peak_bandwidth. Operations with AI < ridge are memory-bound (compute can't be utilized because data isn't arriving fast enough). Operations with AI > ridge are compute-bound (data arrives faster than it can be computed on). For H100 in bf16: ridge ≈ 300 FLOPs/byte. For B200: ridge ≈ 200-300 (depending on precision). For consumer GPUs with relatively more compute than HBM: ridge can be 500-1000. determines which regime applies.

The roofline equation

The ridge point varies dramatically across hardware. A B200 needs ~2× higher arithmetic intensity than H100 to be compute-bound; a Mac Studio’s lower ridge means more ops can be compute-bound there (relatively easier to feed the smaller compute capacity).

Now make it run

The kernel computes arithmetic intensity for common LLM operations and labels each by roofline regime:

       AI = 2N / (2N·2) = 0.5 F/B. Memory-bound. */
    {
        int N = 4096;
        double flops = 2.0 * N;
        double bytes = 2.0 * N * 2;
        report_op("dot product (N=4096, fp16)", flops, bytes, H100_RIDGE);
    }

    /* GEMV: y = M · x. M is m × n, x is n.
       FLOPs: 2·m·n. Bytes: m·n·2 (for M, the dominant input) + n·2 (for x) ≈ 2·m·n.
       AI = 2·m·n / (2·m·n) = 1 F/B. Memory-bound. */
    {
        int m = 4096, n = 4096;
        double flops = 2.0 * m * n;
        double bytes = (double)m * n * 2 + (double)n * 2 + (double)m * 2;
        report_op("GEMV (m=n=4096, fp16)", flops, bytes, H100_RIDGE);
    }

    /* GEMM small: C = A · B. A is m × k, B is k × n. m=n=8 (decode batch), k=4096.
       FLOPs: 2·m·n·k. Bytes: m·k·2 + k·n·2 + m·n·2.
       For m=n=8: bytes dominated by A and B (similar sizes); AI is low. */
    {
        int m = 8, n = 8, k = 4096;
        double flops = 2.0 * m * n * k;
        double bytes = (double)m * k * 2 + (double)k * n * 2 + (double)m * n * 2;
        report_op("GEMM small (m=n=8, k=4096)", flops, bytes, H100_RIDGE);
    }

    /* GEMM medium: m=n=128, k=4096 (still typical batch size for inference). */
    {
        int m = 128, n = 128, k = 4096;
        double flops = 2.0 * m * n * k;
        double bytes = (double)m * k * 2 + (double)k * n * 2 + (double)m * n * 2;
        report_op("GEMM medium (m=n=128, k=4096)", flops, bytes, H100_RIDGE);
    }

    /* GEMM large: m=n=1024, k=4096 (training batch). */
    {
        int m = 1024, n = 1024, k = 4096;
        double flops = 2.0 * m * n * k;
        double bytes = (double)m * k * 2 + (double)k * n * 2 + (double)m * n * 2;
        report_op("GEMM large (m=n=1024, k=4096)", flops, bytes, H100_RIDGE);
    }

    /* Softmax over vocab (V=32K). FLOPs: ~5·V (exp + sum + div). Bytes: 2·V·2 (in, out).
       AI ≈ 5V / 4V = 1.25 F/B. Memory-bound. */
    {
        int V = 32000;
        double flops = 5.0 * V;
        double bytes = 2.0 * V * 2;
        report_op("softmax (V=32K, fp16)", flops, bytes, H100_RIDGE);
    }

    /* LayerNorm: same shape op as softmax. Memory-bound. */
    {

Output:

Roofline analysis — H100 reference
  Peak bf16 FLOPs: 1.00e+15
  Peak HBM BW:     3.35e+12 B/s
  Ridge AI:        298.5 FLOPs/byte

Common LLM operations at typical sizes:
  dot product (N=4096, fp16)      AI=   0.5 F/B  memory-bound
  GEMV (m=n=4096, fp16)           AI=   1.0 F/B  memory-bound
  GEMM small (m=n=8, k=4096)      AI=   4.0 F/B  memory-bound
  GEMM medium (m=n=128, k=4096)   AI=  63.0 F/B  memory-bound
  GEMM large (m=n=1024, k=4096)   AI= 455.1 F/B  COMPUTE-BOUND
  softmax (V=32K, fp16)           AI=   1.2 F/B  memory-bound
  LayerNorm (d=4096)              AI=   1.3 F/B  memory-bound

Read carefully: every LLM operation EXCEPT large GEMM (m, n ≥ 1024) is memory-bound on H100. Dot products, vector ops, softmax, LayerNorm, even small/medium GEMM — all bandwidth-bound, meaning the GPU’s compute units sit idle waiting for HBM.

What this means for LLM systems

The pattern is clear: LLM inference is memory-bandwidth-bound; LLM training is closer to compute-bound. This shapes the entire hardware/software trajectory:

• Hardware: each successive generation increases HBM bandwidth more than peak FLOPs. H100 → B200: 2.5× HBM bandwidth, 4× FLOPs. The ridge has SHIFTED in B200, but bandwidth is still increased aggressively because that’s the bottleneck.
• Quantization: smaller dtypes reduce bytes loaded per element → improve AI for memory-bound ops → real wall-clock speedup. This is why int4 inference is ~3× faster than fp16 even though FLOPs are identical.
• Batching: larger inference batches push GEMMs from GEMV-sized to GEMM-sized, moving them across the ridge from memory-bound to compute-bound.

Next: §21.2 — Tensor cores, fp8, NVLink. The specific hardware features that distinguish H100/B200 from prior GPUs and shape how LLM workloads run.