Follow Hi 你! through position embeddings, LN and QKV, two residual additions, the MLP, vocabulary logits, cross-entropy and AdamW, then trace fixed-parameter generation and incremental decoding. Three diagrams and two interactive walkthroughs distinguish parameters, activations, gradients and context.
Set beside the 2017 paper, the model written in part 25 is the decoder half with cross-attention removed; set beside nanoGPT, its structure is practically identical and every difference is engineering. The part most worth chewing on is the three lines that merge the heads: c_attn lays the small matrices side by side, and view + transpose moves “head” forward — worked through by hand on a two-head example. Along the way: a correction (a bias on q is not wasted), the 6N compute estimate, and how pre-training, SFT and RLHF turn a model that can only continue text into ChatGPT.
Part 24's attention head goes into the model, then the model gets deeper and bigger, and val falls from 2.49 all the way to 1.48. Multi-head attention adds not a single parameter — the gain comes from division of labour; the FFN supplies the only non-linearity; residuals make a deep network start out as a shallow one, LayerNorm normalises along the other direction, and Dropout drops from the branches, never the trunk. One question asked four times — what follows “thou”? — tracks it all: an r that never once follows “thou” in the text falls from first place to 0.001.
softmax(QKᵀ/√dₖ)V — this part earns each of its symbols one at a time. Starting from the crudest option, plain averaging, it goes through “a lower-triangular matmul is a weighted sum” and “softmax with −∞ masks the future” to “scores computed as q·k”: from averaging to attention, the code changes by one line. Encoder and decoder differ by one line too. And the final √dₖ is part 15's variance formula used in a new place.