Follow Hi 你! through position embeddings, LN and QKV, two residual additions, the MLP, vocabulary logits, cross-entropy and AdamW, then trace fixed-parameter generation and incremental decoding. Three diagrams and two interactive walkthroughs distinguish parameters, activations, gradients and context.
The previous three parts derived all twenty-six steps from the loss to every parameter, but that was opening the black box and walking through it. This one puts the box back together: cross-entropy's eight steps compressed into one line, BatchNorm's ten into one, both by the same method — count how many times a variable appears in the forward pass and the backward has that many terms. Then backward() is commented out and the model trains for 200,000 steps: val loss 2.1105, against PyTorch's 2.1106.
Five steps left, no forks, and the first three are a copy of the previous part. What is genuinely new is the last two: why view's backward is simply its own inverse (one test decides it — did the count of numbers change), and why dC[Xb] += demb, a line that looks entirely correct, is wrong. Finish these five and all 4137 parameter gradients have been worked out by hand.
The previous part stopped at logits — the edge of the network, with not one parameter touched in eight steps. This one goes thirteen steps further in, and parameters appear for the first time. BatchNorm's nine steps are the worst stretch in the episode for the same reason softmax forks: a normalisation's numerator has to go build the denominator first. Along the way: the first sum-backward whose coefficient is not 1, and a mistake only 3% wide whose maxdiff looks like rounding error.
The previous part finished deriving the test for tensor calculus; this one applies it eight times over, working back from the loss to the logits. What is worth taking away is not the eight expressions but a way of marking your own work, three operating rules, and one property — a gradient that must be exactly zero — that doubles as a debugging tool.
Part 3 gave the Σ of the multivariable chain rule and part 13 used it to derive the linear layer — but that was one special case. This part turns it into a general method: the addition comes from the total differential, the four shapes differ only in which entries of one Jacobian are zero, and whether to sum or broadcast — and along which axis — reads straight off the two index sets.
A gradient always has the shape of whatever it differentiates, and two rules are enough to derive the backward formulas of a linear layer. The Jacobian of C[X] holds only 0s and 1s — and both properties of the embedding gradient, sparsity and accumulation, are written on that one table.
Automatic differentiation in a hundred lines of Python: a Value class builds the graph, _backward closures hold the local rules, a topological sort does the global scheduling. And why += is the Σ from multi-path summation.
Going from 1 output neuron to 2 adds one step to backpropagation. That step is not a new rule — the multivariable chain rule had it all along.
A 2→2→1 network, and where every single number in three rounds of training comes from. Plus two things you only see once you actually run it.