Day 039 — Attention by Hand

The shape

Scaled dot-product attention, no framework:

Attention(Q,K,V) = softmax(QKᵀ / √dₖ) V

With a causal mask: scores = QKᵀ / √dₖ + mask where mask = 0 allowed, -inf blocked.

Forward

scores = (Q @ K.T) / np.sqrt(d_k) + mask  # (T, T)
weights = softmax(scores, axis=-1)           # (T, T)
out = weights @ V                            # (T, d_k)

Backward — where it burned

softmax Jacobian is the trap:

dL/dscores = weights * (dL/dweights - sum(dL/dweights * weights))

Then:

dQ = dScores @ K / √dₖ
dK = dScores.T @ Q / √dₖ
dV = weights.T @ dScores_V

Numeric check on T=3, d=4 at 1e-4 — passed after fixing a missing / √dₖ on dK.

Reflection

CNNs taught me to trust the grid. Attention taught me to trust the mask. Tomorrow: multi-head + residual + layernorm — same paper, new choreography.

Day 039. Committed.