Set beside the 2017 paper, the model written in part 25 is the decoder half with cross-attention removed; set beside nanoGPT, its structure is practically identical and every difference is engineering. The part most worth chewing on is the three lines that merge the heads: c_attn lays the small matrices side by side, and view + transpose moves “head” forward — worked through by hand on a two-head example. Along the way: a correction (a bias on q is not wasted), the 6N compute estimate, and how pre-training, SFT and RLHF turn a model that can only continue text into ChatGPT.
Part 24's attention head goes into the model, then the model gets deeper and bigger, and val falls from 2.49 all the way to 1.48. Multi-head attention adds not a single parameter — the gain comes from division of labour; the FFN supplies the only non-linearity; residuals make a deep network start out as a shallow one, LayerNorm normalises along the other direction, and Dropout drops from the branches, never the trunk. One question asked four times — what follows “thou”? — tracks it all: an r that never once follows “thou” in the text falls from first place to 0.001.
Starting over on a new dataset: Shakespeare, a 65-character vocabulary, and a bigram baseline at val 2.49 — the control group every later addition gets measured against. These sections hold no new mathematics, but they introduce two things that run through the whole episode: a token is something you define yourself (which is also the real reason for picking Qwen later), and the three shapes (B, T, C). Plus one equivalence worth seeing once: a lookup is a fully connected layer with no bias.
The six-layer network runs, and all four diagnostic plots are in hand. The first three are snapshots of a single step, and all three are green; the one that yields an action item is the fourth — it turns “what should the learning rate be” from a range a sweep can only narrow into a factor you can compute.
The hidden layer is chapter 1's weighted sum, computed for 228,146 samples at once. A tensor is one strip of memory plus a manual: view costs nothing, a transposed tensor cannot be viewed — and hand-written loss matches F.cross_entropy until it returns nan.