Follow Hi 你! through position embeddings, LN and QKV, two residual additions, the MLP, vocabulary logits, cross-entropy and AdamW, then trace fixed-parameter generation and incremental decoding. Three diagrams and two interactive walkthroughs distinguish parameters, activations, gradients and context.
If a larger vocabulary shortens a sequence, why can output computation still increase? Follow parameter counts and ID mappings into continuous and discrete visual representations, walk through eight steps of Token Healing, then examine representation differences through Chinese, numbers, whitespace and JSON/YAML examples. Includes bilingual diagrams and an interactive walkthrough.
What happens after a tokenizer produces IDs? Follow one five-token model through two embedding lookups, a Transformer Block, every lm_head score, and the training loss. Then work through random draws, temperature, top-k and top-p, with three diagrams, two interactive demos and complete reproducible code.
Make exactly 6 with unlimited coins of value 1, 3 and 4. Derive the recurrence from the meaning of dp[x]: why it reads x−c, why worse paths can be discarded, and why six rounds of computation lead to a backtrace of just 6→3→0. Then apply the same reasoning to Unigram's Viterbi encoding.
Unigram updates piece probabilities using expected counts over all valid segmentations, then prunes a candidate vocabulary by removal cost. Derive p, w, Z, q, and C on one corpus, use the resulting model for abcabc's final Viterbi step, and verify SentencePiece byte fallback.
cl100k_base supplies bytes-to-rank/ID entries; minbpe reconstructs their left and right components. Follow hello to see why recovery stops strictly below the target rank, then separate merges, expanded vocab, shuffled base IDs, and raw bytes. Special-token policies select dedicated IDs or ordinary BPE, with complete round-trip checks.
GPT-2's encoder.py maps 256 bytes to reversible character aliases before performing string BPE. Follow Hi 你! through resources, ranks and IDs, nested loops, caching, and stop conditions, then join bytes across tokens before UTF-8 decoding. Includes interactive traces of fourteen official merges and seven teaching-example merges.
Unconstrained BPE learns combinations such as e plus a space or punctuation plus whitespace. GPT-2 first splits text with a regex: chunks share one learned merge table, but merges cannot cross their boundaries. Following four indentation spaces into a three-plus-one split separates regex decisions from learned vocabulary decisions.
A model sees nothing but IDs, and the tokenizer translates between text and IDs. Code points make too big a vocabulary and bytes too long a sequence, so BPE starts from 256 bytes and keeps merging the pair that most often sits together into a new token — and once trained, all that remains is a merge table in order. Encoding takes, at each step, the pair learned earliest, which is why ' th' sits in the table yet goes unused when cutting 'the thin': the order decides the result.
Part 23 said a tokenizer turns text into integers, but not that the first step turns text into bytes. A charset only says which number a character is; an encoding only says which bytes that number becomes — and once they are separated, UTF-8's four templates can be worked out by hand, half a Hanzi has a definite reason for showing as �, and the 256 entries at the bottom of a tokenizer's vocabulary stop being a convention and become a mechanical fact.
Set beside the 2017 paper, the model written in part 25 is the decoder half with cross-attention removed; set beside nanoGPT, its structure is practically identical and every difference is engineering. The part most worth chewing on is the three lines that merge the heads: c_attn lays the small matrices side by side, and view + transpose moves “head” forward — worked through by hand on a two-head example. Along the way: a correction (a bias on q is not wasted), the 6N compute estimate, and how pre-training, SFT and RLHF turn a model that can only continue text into ChatGPT.
Part 24's attention head goes into the model, then the model gets deeper and bigger, and val falls from 2.49 all the way to 1.48. Multi-head attention adds not a single parameter — the gain comes from division of labour; the FFN supplies the only non-linearity; residuals make a deep network start out as a shallow one, LayerNorm normalises along the other direction, and Dropout drops from the branches, never the trunk. One question asked four times — what follows “thou”? — tracks it all: an r that never once follows “thou” in the text falls from first place to 0.001.
softmax(QKᵀ/√dₖ)V — this part earns each of its symbols one at a time. Starting from the crudest option, plain averaging, it goes through “a lower-triangular matmul is a weighted sum” and “softmax with −∞ masks the future” to “scores computed as q·k”: from averaging to attention, the code changes by one line. Encoder and decoder differ by one line too. And the final √dₖ is part 15's variance formula used in a new place.
Starting over on a new dataset: Shakespeare, a 65-character vocabulary, and a bigram baseline at val 2.49 — the control group every later addition gets measured against. These sections hold no new mathematics, but they introduce two things that run through the whole episode: a token is something you define yourself (which is also the real reason for picking Qwen later), and the three shapes (B, T, C). Plus one equivalence worth seeing once: a lookup is a fully connected layer with no bias.
The previous three parts derived all twenty-six steps from the loss to every parameter, but that was opening the black box and walking through it. This one puts the box back together: cross-entropy's eight steps compressed into one line, BatchNorm's ten into one, both by the same method — count how many times a variable appears in the forward pass and the backward has that many terms. Then backward() is commented out and the model trains for 200,000 steps: val loss 2.1105, against PyTorch's 2.1106.
Five steps left, no forks, and the first three are a copy of the previous part. What is genuinely new is the last two: why view's backward is simply its own inverse (one test decides it — did the count of numbers change), and why dC[Xb] += demb, a line that looks entirely correct, is wrong. Finish these five and all 4137 parameter gradients have been worked out by hand.
The previous part stopped at logits — the edge of the network, with not one parameter touched in eight steps. This one goes thirteen steps further in, and parameters appear for the first time. BatchNorm's nine steps are the worst stretch in the episode for the same reason softmax forks: a normalisation's numerator has to go build the denominator first. Along the way: the first sum-backward whose coefficient is not 1, and a mistake only 3% wide whose maxdiff looks like rounding error.
The previous part finished deriving the test for tensor calculus; this one applies it eight times over, working back from the loss to the logits. What is worth taking away is not the eight expressions but a way of marking your own work, three operating rules, and one property — a gradient that must be exactly zero — that doubles as a debugging tool.
Part 3 gave the Σ of the multivariable chain rule and part 13 used it to derive the linear layer — but that was one special case. This part turns it into a general method: the addition comes from the total differential, the four shapes differ only in which entries of one Jacobian are zero, and whether to sum or broadcast — and along which axis — reads straight off the two index sets.
The six-layer network runs, and all four diagnostic plots are in hand. The first three are snapshots of a single step, and all three are green; the one that yields an action item is the fourth — it turns “what should the learning rate be” from a range a sweep can only narrow into a factor you can compute.
Initialisation only governs step 0; BatchNorm governs every step — vary σ_W by a factor of a thousand and the output after BN is identical. The price is that it becomes the one operation in the network where samples affect each other, which is why inference needs its own statistics, and why model.eval() exists.
The previous part borrowed σ_z = √n · σ_W · σ_x and moved on. This one derives it from the definition of expectation: three rules of arithmetic, why the cross term vanishes, why it is variance and not standard deviation that adds — and why diagnosing a network means watching the variance rather than the mean or the maximum.
An initial loss of 17 and six in ten neurons dead on arrival — because the standard deviation is amplified once at each of two matrix products. The fix is four lines, but what actually travels is the checklist of what to check in what order, and why Kaiming loses to a guessed number on this particular network.
A gradient always has the shape of whatever it differentiates, and two rules are enough to derive the backward formulas of a linear layer. The Jacobian of C[X] holds only 0s and 1s — and both properties of the embedding gradient, sparsity and accumulation, are written on that one table.
The model fits parameters on train; you fit hyperparameters on val — which is why test needs a third slice. Split by name, not by sample, or the validation set is contaminated on the spot. Tuned, val lands at 2.1752 — and on the embedding plot the vowels have gathered on their own.
A gradient from 32 samples is wrong, yet 820× faster than the full batch and lands a lower loss. The learning rate comes from a sweep: even spacing in the exponent is even ratios in the rate — and why that U-curve can only rule out, never select.
The hidden layer is chapter 1's weighted sum, computed for 228,146 samples at once. A tensor is one strip of memory plus a manual: view costs nothing, a transposed tensor cannot be viewed — and hand-written loss matches F.cross_entropy until it returns nan.
One-hot makes a as related to e as it is to q; structure only appears once 27 dimensions are squeezed into 2. The table grows 729 → 531,441 while the first layer shrinks 8,100 parameters → 600 — and a dot product is not a cosine.
First build a complete language model with no neural network at all, purely by counting (loss 2.4544), then rebuild it with one-hot, matrix multiplication and softmax (2.46). The two roads meet — which answers what a neural network is actually learning.
Cat-versus-dog and predicting the next character are the same problem, differing only in that the class count goes from 2 to 27. The three parts that difference requires: one-hot, softmax, cross-entropy — and why cross-entropy is not a fancier squared error.
Connecting the parts from the last two: a loss function gathers the outputs into one number, and a three-part loop actually drives it down. Plus what happens when you deliberately skip zeroing the gradients — across ten random seeds.
Wrapping Value into Neuron / Layer / MLP. Where the parameter count comes from, how sz's sliding window lines the dimensions up, and why a three-line loop is the entire forward pass.
Automatic differentiation in a hundred lines of Python: a Value class builds the graph, _backward closures hold the local rules, a topological sort does the global scheduling. And why += is the Σ from multi-path summation.