2.1752: The Ruler Comes Before the Tuning

The previous part pushed the loss down to 2.33, past bigram. But every one of those numbers was computed on the training set — they only answer how much the model memorised.

This part builds the ruler first, then uses it to tune. Nothing clever here; it is simply turning one number into two — and it is the foundation everything after it stands on.

One number is not enough

Recall the 5-sample overfitting experiment from the previous part: loss = 0.001965, apparently flawless, and that model was useless on any new name.

The same mechanism is running on the current model at a small scale, and it is about to get worse:

Triple the parameters and the capacity to memorise rises with them. What follows is a situation where the training loss keeps falling and everything looks like progress, while the real ability is already going backwards — and one number cannot show you that at all.

The order has to be: build the evaluation first, then grow the model. Done the other way round, every experimental conclusion that follows is worthless.

Three sets, and who gets to see them

Why val and test must be separate

If both are “data never trained on”, why not just one?

Look at how tuning actually goes:

n_embd=2,  hidden=100  → val 2.31
n_embd=10, hidden=100  → val 2.24
n_embd=10, hidden=300  → val 2.19
n_embd=10, hidden=500  → val 2.20   ← worse, back up
... fifty attempts, keep whichever had the lowest val

Every time you look at val, a little of val leaks into the model. After fifty attempts, the chosen hyperparameters are partly genuinely good and partly just lucky on that particular 10%. The val figure is optimistic as a result.

   train        →        val        →        test
model fits params    you fit hyperparams    nobody fits

The model’s parameters are learned from train; your hyperparameters are learned from val. You are part of this system too.

The discipline is: test is used once. Look at test, dislike the result, go back and change the model, look again — test has just become a second val.

Kaggle works on exactly this mechanism: during the competition you only see the public leaderboard (val), and the private one (test) is revealed at the last moment. Plenty of people grind the public board high and then drop on the private reveal — that is overfitting to val.

Why 80/10/10

Nothing sacred; it is convention. The criterion: val and test must be large enough for the loss figure to be stable.

10% = 3,203 names ≈ 20,000 samples, which is plenty. With only 1,000 records, 10% is 100 and val would jitter too much to use — then it becomes 60/20/20 or cross-validation. The more data, the smaller the validation share can be: ImageNet has 1.28 million images and a validation set of only 4%.

Split by name, not by sample

emma produces five samples:

[.  .  .] → e
[.  .  e] → m
[.  e  m] → m
[e  m  m] → a
[m  m  a] → .

Split those samples randomly and it might come out like this:

train: [. . .] → e     [. e m] → m     [m m a] → .
val:   [. . e] → m     [e m m] → a

The model saw the beginning and end of emma while training, then gets examined on a slice from the middle — that is no longer “data it has not seen”.

contaminated validation set  ⟹  val loss too low  ⟹  you think the model is good

So the 32,033 names get split into three piles first, and build_dataset is called on each.

General principle: split at the smallest independent unit. Here it is the name; for medical imaging it is the patient (scans of one patient must not straddle sets); for time series it is by time, never shuffled.

The code, and why shuffle is necessary

import random
words = open('names.txt').read().splitlines()   # re-read from the file each time
random.seed(42)
random.shuffle(words)

n1 = int(0.8 * len(words))     # 25626
n2 = int(0.9 * len(words))     # 28829

Xtr,  Ytr  = build_dataset(words[:n1])      # 182,625 samples
Xdev, Ydev = build_dataset(words[n1:n2])    # 22,655
Xte,  Yte  = build_dataset(words[n2:])      # 22,866

shuffle randomises a list’s order in place, returning None — writing b = random.shuffle(a) gives you None.

Why it is needed: names.txt is sorted by frequency — emma and olivia come first and the tail is all obscure oddities. Split without shuffling:

All three sets must come from the same distribution — that is the precondition for splitting data at all.

random.seed(42) turns “random” into reproducibly random. Without it, every re-run of the split during tuning differs and val moves from 2.19 to 2.23 — and you cannot tell whether the hyperparameter or the reshuffle did it.

⚠️ Every X/Y in the training loop has to become Xtr/Ytr. Miss one and the model trains on the full dataset, the validation set is void on the spot, and nothing raises an error.

How to read two numbers

The gap (how far train and val differ) answers “is it memorising”; the level (the absolute loss) answers “is it learning well enough”. The two are unrelated, and combined they give four cases:

① and ② have exactly the same train/val gap, yet one is underfitting and the other is the best state there is. The gap alone cannot separate them.

The extreme counter-example: a model with three parameters scores train 3.29 / val 3.29, a gap of zero — and it is obviously the worst model possible.

Judging “level” needs a reference

That last row is the key one: the same MLP structure can reach 2.17, so there is 0.16 still unextracted from this architecture.

The full test for underfitting takes two conditions: train ≈ val (not memorising) plus “a larger model is known to do better” (there is room in the level). Neither alone suffices.

The gap answers “can I grow it”; the absolute value answers “should I”.

Strictly, train ≈ val only proves the model’s capacity is not yet filled by the data, so growing it is safe. Whether growing it helps is another matter — that has to be tried.

Measured

train 2.3320   val 2.3313

val is 0.0007 below train — each set has its own randomness and that difference sits entirely within the noise. They are essentially the same number.

The conclusion: 3,481 parameters against 182,625 training samples do not even have the capacity to memorise, so the model is forced to learn real patterns. Growing it is safe.

This also confirms the split is clean. Had it been split by sample rather than by name, the validation set would be contaminated and val would sit noticeably below train. The two numbers being this close says the split was right.

Four knobs

Depth is not on this list. Adding a third layer means restructuring the code, and without the next episode’s initialisation tricks, going deeper will probably be worse — leaving you unable to tell “depth is bad” from “the initialisation is wrong”. Control the variables first.

Depth, width and embedding size are three different things

X (N,3) ─C[X]→ (N,3,10) ─flatten→ (N,30) ─@W1+b1→ (N,200) ─@W2+b2→ (N,27)
   ↑              ↑                            ↑                    ↑
block_size=3   n_embd=10                 n_hidden=200         27, locked

The two bottlenecks can sit in different places: with n_embd=1, n_hidden=1000 the input end is already choked and no amount of hidden width rescues it. So they get separate names and separate tuning.

Against real models:

Real models grow wide and deep together. Both extremes are bad: 100 layers × width 16 is too thin (an information bottleneck); 2 layers × width 100,000 explodes the parameter count for little return.

Depth pays better, but it takes a whole toolkit to collect. Before 2012 almost nobody could train a network more than a few layers deep — not for lack of ideas, but because it would not train. ResNet (2015) opened a motorway for gradients with residual connections, and only then did 152 layers become possible.

The universal approximation theorem says two layers wide enough can approximate any continuous function — but “wide enough” may mean exponentially wide, where a deep network needs only linearly many.

The experiment log

⚠️ This comparison is not clean — the step counts differ. Strictly only the last row is reliable. A fair comparison would re-run 100 and 300 at 130k steps too.

The diagnostic rule

When hid=300 took val from 2.2340 up to 2.2823, the first instinct says “more parameters, so it overfitted” — and that is wrong. Look at train:

train 2.2256 → 2.2338      ← train got worse too

The two prescriptions are opposites, and getting the direction wrong wastes the effort entirely. Confirmed: after another 30k steps, train 2.2184 / val 2.2665 — both falling, so it was undertraining.

Three lessons

1. Hyperparameters have a sweet spot; bigger is not better. 100 too small, 300 too large, 200 just right.
2. A larger model needs more steps. Changing the capacity without changing the step count makes the comparison unfair.
3. Change one thing at a time, or you cannot tell what did it; and reset the parameters for every run.

Converting steps to epochs: 130,000 × 32 ÷ 182,625 ≈ 22.8 epochs.

Modern large models report steps and token counts and rarely mention epochs — because with different dataset sizes, “one epoch” means wildly different things. GPT-3 did not even complete one.

Plotting the embedding

Set n_embd back to 2, retrain, and the whole table can simply be drawn on paper:

plt.scatter(C[:,0].data, C[:,1].data, s=200)
for i in range(27):
    plt.text(C[i,0].item(), C[i,1].item(), itos[i], ha="center", va="center")

Distance = substitutability. What the model learned is not the concept “vowel”. It learned that a and e occur in similar places and are followed by similar things, so giving them nearby coordinates makes the prediction better. The classification is a by-product, never the goal.

Part 9 ended on “structure is not designed; it is forced out by having too few dimensions”. Here it is cashed in. The figure was deliberately withheld then — the network had never been trained, so drawing it would have been a fabrication.

Why C gets adjusted too

The common misreading: C is an input and should not be changed. Wrong.

The key identity:

emb = onehot(X) · C

That is one ordinary x @ W. Because a one-hot holds a single 1, the matrix multiplication degenerates into “take row number k”. So the real structure has three layers:

one-hot (27) ─@C→ (n_embd) ─@W1+b1→ (n_hidden) ─@W2+b2→ (27)
              ↑
        C is that layer's weight matrix

On what grounds would W1 be trained and this not? It is not written as a one-hot multiplication only because 99% of that computation would be multiplying by zero.

Fix C in place and there are two options: fix it to one-hot, and every pair of characters is the same distance apart — back to V2; fix it to random 2-D values, and a may sit right next to z — a wrong prior, worse than none. There is no third option — you cannot hand-write a correct C.

How the gradient reaches C

∂L/∂C[v] = sum the emb gradients from every position in the batch that used character v

Two properties: sparse — rows that were not used get a gradient of exactly 0, and since a batch touches at most 96 positions, perhaps only a dozen of the 27 rows are non-zero; additive — a character appearing more than once has its gradients summed, and in the sample [5,13,13] where m appears twice, measured: ∂L/∂C[13] = [+1.472,−0.681] + [+2.315,−0.284] = [+3.787,−0.965].

The second property is exactly the point of part 4’s a + a experiment — a variable used k times in the graph accumulates its gradient k times, which is why _backward has to use +=.

And the first is q’s predicament: a rare letter is touched once in a very long while, so it is still sitting near where it started on the scatter plot.

The whole chain, walked once — six steps forward, six back, ending on C:

One step in the backward half is worth watching: passing through tanh multiplies the gradient by (1 − h²), and across this batch only 50% survives on average. That is one of the threads left for the next episode; the more layers, the worse the decay — and it is the direct reason depth is off the table for now.

Sampling: generating new names

for _ in range(20):
    out = []
    context = [0] * block_size                    # start from "..."
    while True:
        emb    = C[torch.tensor([context])]
        h      = torch.tanh(emb.view(1, -1) @ W1 + b1)
        logits = h @ W2 + b2
        probs  = F.softmax(logits, dim=1)
        ix = torch.multinomial(probs, num_samples=1, generator=g).item()
        context = context[1:] + [ix]               # feed the new character back in
        if ix == 0: break
        out.append(itos[ix])

Sampling itself was covered thoroughly in part 8; only three details matter here. emb.view(1, -1) — the batch size is 1, and the previous batch’s “never hard-code a shape” pays off again. multinomial draws by probability; with argmax all twenty names would be identical. And context = context[1:] + [ix] is autoregression — GPT generates text with this same loop, only with the MLP swapped for a Transformer.

The actual output:

carmahela  jhevi  kimrie  taty  sacaysa  jazhnen  amerynci  aqui
nermara  chaiir  kaleigh  ham  porriquint  sulin  alianni
waythoniearynix  kaellinsley  dae  oiq  gian

Where the progress is, and where the failures are

Against part 8’s bigram output — cexze, momasurailezitynn, konimittain:

• kaleigh — it learned the -eigh combination. Getting i from e and g from i has no justification from the preceding letter alone; bigram could never learn it.
• carmahela, nermara — vowels and consonants alternate naturally, with none of cexze’s three-consonant pile-ups.
• waythoniearynix — far too long, but locally correct throughout (way/thon/nie/ary). With three characters of context, the model has no idea it has already written fifteen letters.
• oiq — a failure. q should be followed by u and the model never learned it, because row 17 of C has barely been updated — precisely the point that flew to the top of the scatter plot.

The new bottleneck is not that the model is too small; it is that it cannot see far enough. But every increment of block_size adds n_embd to the input width — ten characters would mean a 100-dimensional input, and the approach does not scale.

This is exactly why the Transformer exists: attention handles context of any length without the parameter count growing linearly with it.

V3 ends here:

The plan called for about 2.2; the result is 2.1752.

What actually travels out of these four parts (9–12), in order of importance: the tensor memory model (indispensable once tensors get high-dimensional), the train/val/test split (without it, all tuning is blind), the tiny-slice overfitting check (two minutes to separate “code bug” from “strategy problem”), searching in log space (a general method, not just for learning rates), mini-batching, and representation learning.

One part sits between here and that episode. Across these four, C has been treated as a matrix you look things up in — but it is a parameter, so it must have a gradient. The next part works out what differentiating an index actually computes to.

After that, the next episode fixes three things, all of them measured already: the initial loss that should be 3.30 and is 17; the share of values in h pinned at ±1, far above where it should be; and the gradient losing half its magnitude passing through tanh. Only after that does deepening the network become defensible.