If a larger vocabulary shortens a sequence, why can output computation still increase? Follow parameter counts and ID mappings into continuous and discrete visual representations, walk through eight steps of Token Healing, then examine representation differences through Chinese, numbers, whitespace and JSON/YAML examples. Includes bilingual diagrams and an interactive walkthrough.
What happens after a tokenizer produces IDs? Follow one five-token model through two embedding lookups, a Transformer Block, every lm_head score, and the training loss. Then work through random draws, temperature, top-k and top-p, with three diagrams, two interactive demos and complete reproducible code.
Starting over on a new dataset: Shakespeare, a 65-character vocabulary, and a bigram baseline at val 2.49 — the control group every later addition gets measured against. These sections hold no new mathematics, but they introduce two things that run through the whole episode: a token is something you define yourself (which is also the real reason for picking Qwen later), and the three shapes (B, T, C). Plus one equivalence worth seeing once: a lookup is a fully connected layer with no bias.
Five steps left, no forks, and the first three are a copy of the previous part. What is genuinely new is the last two: why view's backward is simply its own inverse (one test decides it — did the count of numbers change), and why dC[Xb] += demb, a line that looks entirely correct, is wrong. Finish these five and all 4137 parameter gradients have been worked out by hand.
A gradient always has the shape of whatever it differentiates, and two rules are enough to derive the backward formulas of a linear layer. The Jacobian of C[X] holds only 0s and 1s — and both properties of the embedding gradient, sparsity and accumulation, are written on that one table.
One-hot makes a as related to e as it is to q; structure only appears once 27 dimensions are squeezed into 2. The table grows 729 → 531,441 while the first layer shrinks 8,100 parameters → 600 — and a dot product is not a cosine.