What happens after a tokenizer produces IDs? Follow one five-token model through two embedding lookups, a Transformer Block, every lm_head score, and the training loss. Then work through random draws, temperature, top-k and top-p, with three diagrams, two interactive demos and complete reproducible code.
softmax(QKᵀ/√dₖ)V — this part earns each of its symbols one at a time. Starting from the crudest option, plain averaging, it goes through “a lower-triangular matmul is a weighted sum” and “softmax with −∞ masks the future” to “scores computed as q·k”: from averaging to attention, the code changes by one line. Encoder and decoder differ by one line too. And the final √dₖ is part 15's variance formula used in a new place.
The previous part finished deriving the test for tensor calculus; this one applies it eight times over, working back from the loss to the logits. What is worth taking away is not the eight expressions but a way of marking your own work, three operating rules, and one property — a gradient that must be exactly zero — that doubles as a debugging tool.
First build a complete language model with no neural network at all, purely by counting (loss 2.4544), then rebuild it with one-hot, matrix multiplication and softmax (2.46). The two roads meet — which answers what a neural network is actually learning.
Cat-versus-dog and predicting the next character are the same problem, differing only in that the class count goes from 2 to 27. The three parts that difference requires: one-hot, softmax, cross-entropy — and why cross-entropy is not a fancier squared error.