#Overfitting

2026-09-24·—READS·—LIKES·Learning AI with AI

How the r After “thou” Disappeared: Growing One Attention Head into a GPT

Part 24's attention head goes into the model, then the model gets deeper and bigger, and val falls from 2.49 all the way to 1.48. Multi-head attention adds not a single parameter — the gain comes from division of labour; the FFN supplies the only non-linearity; residuals make a deep network start out as a shallow one, LayerNorm normalises along the other direction, and Dropout drops from the branches, never the trunk. One question asked four times — what follows “thou”? — tracks it all: an r that never once follows “thou” in the text falls from first place to 0.001.

2026-09-06·—READS·—LIKES·Learning AI with AI

2.1752: The Ruler Comes Before the Tuning

The model fits parameters on train; you fit hyperparameters on val — which is why test needs a third slice. Split by name, not by sample, or the validation set is contaminated on the spot. Tuned, val lands at 2.1752 — and on the embedding plot the vowels have gathered on their own.