Here is the atom. Every score in this book — the similarity between two embeddings, the QKᵀ entry inside attention, the distance computation inside HNSW forward ref Hierarchical Navigable Small World: the dominant graph index for approximate nearest-neighbor search. Its inner loop is a dot product. Ch. 25. The reason we drill the dot product now is that HNSW traversal calls it millions of times — making it the hottest kernel in vector search. — bottoms out in this single operation.

You know the mechanics. What’s worth re-seizing is the geometry, because it’s what makes the dot product the universal similarity measure:

Read that right-to-left and it tells a story: the dot product is large and positive when two vectors point the same way (cos θ → 1), zero when orthogonal core term Perpendicular — dot product exactly zero. In ML we read it as 'these two vectors share no component,' i.e. carry independent information. Then → now: same linear-algebra meaning, but Ch. 6 reveals a shock — in high dimensions, almost every pair of random vectors is nearly orthogonal. That fact drives why embeddings and quantization behave the way they do. (θ = 90°), and negative when they oppose. “Similarity” in an embedding space is just this angle. When you strip out magnitude — divide by both lengths — you get cosine similarity core term ⟨a,b⟩ / (‖a‖‖b‖) — the dot product with magnitude divided out, so it measures angle alone, range [−1, 1]. The default similarity metric for text embeddings today. 'Dot vs cosine vs L2' (Ch. 9 / Ch. 25) is a choice about whether magnitude should count. , the metric most text-embedding systems use by default.

Now make it run — scalar first, then SIMD

A toy Python dot product teaches the math but hides the thing you actually care about: how the machine does it fast. We’ll always show both. Here is the honest scalar version, then the vectorized one.

static float dot_scalar(const float* a, const float* b, int n) {
    float acc = 0.0f;
    for (int i = 0; i < n; i++) acc += a[i] * b[i];
    return acc;
}

One multiply-add hardware term Fused Multiply-Add: computes a·b + c as one instruction with a single rounding — faster and more numerically accurate than a separate multiply then add. Then → now: separate MUL+ADD in your era; fused since ~2013 (Haswell). The accuracy gain matters for the long accumulations in Ch. 3. per element, fully serial. The CPU can do eight of these at once. The AVX version packs eight floats per register and uses one fused multiply-add intrinsic:

    __m256 acc = _mm256_setzero_ps();   /* 8 float lanes, all 0.0 — running sums */
    int i = 0;
    for (; i + 8 <= n; i += 8) {
        __m256 va = _mm256_loadu_ps(a + i); /* 8 floats of a → register */
        __m256 vb = _mm256_loadu_ps(b + i); /* 8 floats of b → register */
        acc = _mm256_fmadd_ps(va, vb, acc); /* 8× (aᵢ·bᵢ + accᵢ), one instr, one rounding */
    }
    /* ---- horizontal reduction: 8 lanes → 1 scalar ---- */
    __m128 lo = _mm256_castps256_ps128(acc);   /* lanes 0–3 (free reinterpret) */
    __m128 hi = _mm256_extractf128_ps(acc, 1); /* lanes 4–7 */
    lo = _mm_add_ps(lo, hi);                   /* {0+4, 1+5, 2+6, 3+7} → 4 sums */
    lo = _mm_hadd_ps(lo, lo);                  /* pairwise → 2 sums */
    lo = _mm_hadd_ps(lo, lo);                  /* pairwise → 1 sum in lane 0 */
    float acc8 = _mm_cvtss_f32(lo);            /* lane 0 → plain float */

    for (; i < n; i++) acc8 += a[i] * b[i];    /* tail when n not divisible by 8 */
    return acc8;
}

Two things to notice, because they recur in every kernel in this book. (1) The accumulator is a vector, not a scalar — you keep eight independent running sums and only collapse them at the very end. That “carry partial results, reduce once” pattern is exactly the online-reduction trick callback / forward ref Carry running statistics through a tiled computation, reduce once at the end. Here: 8 partial sums. In FlashAttention (Ch. 13): a running max + sum for softmax. Same idea you'll see in FlashAttention. The horizontal sum here is a reduction; the only question in harder cases is whether the reduction is associative (this one is; softmax isn't, which is why it needs more care). behind FlashAttention (Ch. 13). (2) The horizontal sum is awkward on purpose — SIMD lanes are great at independent work, bad at talking to each other. That tension drives a lot of kernel design.

Depth pass — every named intrinsic, taken apart

The prototype above names a stack of intrinsics like they’re self-evident. They’re not. Before any instruction, you need to know what it operates on.

A modern x86 CPU has a set of extra-wide registers — physically, slots of silicon on the chip — called the AVX registers, each 256 bits wide. That’s the whole story behind the type name __m256:

This is the mental upgrade from your scalar past: a float acc is one number in one register. A __m256 acc is eight numbers in one wide register, and an instruction like “add” hits all eight in the same clock. The variants you’ll meet are just other widths and types:

Every Intel intrinsic (a C function that compiles to one CPU instruction) follows a rigid naming grammar. Once you can parse the name, you can read code you’ve never seen. Take the one that looked like noise above:

So _mm256_setzero_ps = “256-bit, set to zero, as floats” → eight zero lanes. _mm256_loadu_ps = “256-bit, load unaligned, as floats” → copy 8 floats from memory into a register (the u means the address needn’t be 32-byte-aligned; the aligned version _mm256_load_ps is marginally faster but crashes on a misaligned pointer). You are now reading intrinsics as sentences, not memorizing them.

What fmadd actually computes, lane by lane

The fused multiply-add takes three register arguments and computes, for every lane i simultaneously:

This is why the accumulator is itself a __m256: each of the eight lanes carries its own running partial sum. Lane 0 accumulates a₀b₀ + a₈b₈ + a₁₆b₁₆ + …, lane 1 accumulates a₁b₉…, and so on. Eight independent dot products of every-8th element, running in parallel. They only get combined at the very end.

That “very end” — the horizontal sum, dismantled

After the loop you hold eight partial sums in one register and need their total — a single scalar. SIMD hardware is built for vertical ops (lane i of one register with lane i of another) and is deliberately clumsy at horizontal ops (combining lanes within one register). That clumsiness is the whole reason this tail looks weird. Walk it one instruction at a time:

The fully annotated, compiled source — every line accounted for — is in code/ch01/dot_avx.c. It is the same kernel the Astro build verifies on every build.