How the r After “thou” Disappeared: Growing One Attention Head into a GPT

Part 24 ended: put the head into the model with its scaling, and the loss finally starts coming down from 2.49.

This part walks that stretch in four steps: one head in the model → multi-head + FFN → going deeper (residuals + LayerNorm) → scaling up + Dropout. After each step the model gets the same question, and the answers show what changed.

The question: what follows “thou”?

Part 24 opened with the bigram giving identical predictions for hou, you and thou, because it only sees the final u. First, the answer key — count what actually follows each context across the whole of Shakespeare:

text[i + len(ctx)]  for every position i where  text.startswith(ctx, i)

🔑 An r never follows “thou” anywhere in the text (text.count('thour') == 0). This comes back repeatedly: the higher a model ranks r, the more it is looking only at the last two letters.

Step one: a single head in the model

The head part 24 demonstrated in a notebook, wrapped as a class:

class Head(nn.Module):
    def __init__(self, head_size):
        super().__init__()
        self.key   = nn.Linear(n_embd, head_size, bias=False)
        self.query = nn.Linear(n_embd, head_size, bias=False)
        self.value = nn.Linear(n_embd, head_size, bias=False)
        self.register_buffer('tril', torch.tril(torch.ones(block_size, block_size)))

    def forward(self, x):
        B, T, C = x.shape
        k, q = self.key(x), self.query(x)                               # (B,T,hs)
        wei = q @ k.transpose(-2, -1) * k.shape[-1] ** -0.5             # (B,T,T)
        wei = wei.masked_fill(self.tril[:T, :T] == 0, float('-inf'))
        wei = F.softmax(wei, dim=-1)
        return wei @ self.value(x)                                      # (B,T,hs)

Almost the same as the notebook, with three new things.

register_buffer: not trained, but travels with the model

tril is an 8×8 lower-triangular “mask template”, built once in __init__. Why not just write self.tril = ...? model.to(device) only moves what it knows about. A check on a Mac:

d = Demo().to('mps')
d.lin.weight.device    # mps:0   ← the parameter moved
d.tril_buf.device      # mps:0   ← so did the buffer
d.tril_plain.device    # cpu     ← the plain attribute stayed on the CPU
d.state_dict().keys()  # tril_buf, lin.weight, lin.bias — tril_plain is nowhere

A plain attribute stays on the CPU while wei lives on the GPU, and masked_fill fails with a device mismatch.

tril sits in the middle column: a fixed rule, not something to learn, but part of the model. The running_mean / running_var hand-written in part 16 are buffers in PyTorch’s own BatchNorm too.

tril[:T, :T]: the input isn’t always 8 characters

The template is built at full length, but generation starts with T = 1, then T = 3, and so on. Slicing the top-left T×T corner makes it match wei.

Scale by k.shape[-1], not C

Note ⑥ derived it: divide by the dimension of the dot product, which is head_size. The video writes C ** -0.5 (n_embd) — a small slip that the repository later changed to k.shape[-1]. With one head the two happen to be equal (both 32); with several heads they are not.

Two hyperparameters change

max_iters 3000 → 5000 (a more complex model); learning_rate 1e-2 → 1e-3. Attention contains a softmax, and part 24’s note ⑥ showed how weights swing hard to one side once scores grow — the bigram, being just a table, could take 1e-2.

Parameters: token table 65×32 = 2080 + position table 8×32 = 256 + attention 3×32×32 = 3072 + lm_head 32×65 + 65 = 2145 = 7553. Karpathy’s val drops from 2.49 to about 2.40 — the first real gain from attention.

Step two: multi-head, not one parameter more

Where things stand: one row of weights, one point of view

With one head, each position has one row of attention weights, summing to 1. To predict what follows hou, the u needs two things at once: “it starts with h” and “I am u”. With part 24’s v (v_h = [10, 0], v_o = [0, 10], v_u = [5, 5]):

all on h:     [0.80, 0.02, 0.18]  → out_u = [8.87, 1.13]   has the start, little of itself
all on self:  [0.05, 0.10, 0.85]  → out_u = [4.75, 5.25]   has itself, little of the start
half each:    [0.50, 0.00, 0.50]  → out_u = [7.50, 2.50]   ← both added into the same numbers, blurred

The fix: more sets of W_q W_k W_v, results side by side

Head A is part 24’s (its u row looks at the start). Suppose head B has learned to look at itself: u row [0.05, 0.10, 0.85] (the h and o rows as in head A), with its own v: h = [2, 0], o = [0, 2], u = [0, 6]. Each head computes on its own, then the results are concatenated along the last dimension:

         head A (the start)    head B (itself)
h   [ 10.00,  0.00  |   2.00,  0.00 ]
o   [  2.69,  7.31  |   0.54,  1.46 ]
u   [  8.87,  1.13  |   0.10,  5.30 ]

The u row now holds both kinds of information, in separate places, without interfering.

★ Derivation: why concatenate rather than add

A head’s output is always followed by a Linear (the FFN below, or the proj added in section 8). Split that Linear’s weight by columns into two blocks, W = [W_A W_B], and apply it to the concatenation:

W · [a ; b] = W_A · a + W_B · b

Concatenating and then applying a Linear is the same as giving each head its own block of weights and adding. Add first instead:

W · (a + b) = W · a + W · b

Now both heads are forced to share one W — the next layer cannot tell which part came from “look at the start” and which from “look at yourself”, and has to treat them alike. Concatenation leaves the question of how to combine the heads to the layers that follow.

Code: ModuleList and cat(dim=-1)

class MultiHeadAttention(nn.Module):
    def __init__(self, num_heads, head_size):
        super().__init__()
        self.heads = nn.ModuleList([Head(head_size) for _ in range(num_heads)])

    def forward(self, x):
        return torch.cat([h(x) for h in self.heads], dim=-1)

self.sa_heads = MultiHeadAttention(4, n_embd // 4)    # 4 heads × 8 dims, back to 32

• The list comprehension runs Head.__init__ afresh each time → four heads with independent parameters, initialised and trained separately
• It must be nn.ModuleList: modules in a plain Python list are invisible to nn.Module → their parameters never reach model.parameters() → the optimiser never updates them and .to(device) never moves them. The same logic as register_buffer: PyTorch manages only what has been registered
• cat(dim=-1) joins along the last dimension only: for each “sequence b, position t”, the heads’ vectors at that position go end to end — out[b, t, 0:8] is head 0, out[b, t, 8:16] is head 1, and so on. Joining along dim=1 would give (B, 2T, …), treating head 1 as extra positions; along dim=0, as extra sequences

Not one parameter more

7553 → 7553, and Karpathy’s val goes from about 2.40 to about 2.28. The gain is pure division of labour.

How the head count and head size are chosen

By convention n_head × head_size = n_embd: pick the number of heads, and n_embd must divide evenly (nanoGPT has assert n_embd % n_head == 0). Too few heads, too few ways of attending; too many, and each head’s q·k has too few dimensions.

🔑 In practice it runs the other way: fix 64 or 128 dimensions per head first, then let n_embd decide how many heads. Bigger models get more heads; the size of each head barely changes.

Nor is the equation a law: section 8 adds an output projection proj after the concatenation, and with it the concatenated width can be anything — it is projected back to n_embd (Gemma 7B: n_embd 3072, 16 heads × 256 = 4096, projected back to 3072).

Multi-head attention comes from Google’s 2017 paper Attention Is All You Need (Vaswani and seven co-authors), whose footnote credits Noam Shazeer with multi-head attention, scaled dot-product attention (that √dₖ) and the parameter-free position representation. The paper’s argument: a single head averages away information from different subspaces; in its ablations a single head is 0.9 BLEU worse than the best setting, and too many heads also hurt — a trade-off found by experiment.

Step three: the FFN, the only non-linearity

Where things stand: mixing, but no non-linearity

Part 19 sorted a network’s layers into three kinds — mixing, non-linearity, normalisation — and the first two can never be dropped. After step two the model has attention mixing across positions and lm_head mixing across dimensions — and no non-linearity at all. Once attention’s weights are fixed, its output is a weighted sum of v, with no element-wise bend anywhere along the feature dimension.

And a stack of purely linear layers is one layer:

W₂(W₁x + b₁) + b₂ = (W₂W₁)·x + (W₂b₁ + b₂) = W′x + b′

A combined judgement such as “it starts with h and the middle is ou → probably house” is beyond a linear combination.

The fix: FeedForward, the kind of MLP from part 5

class FeedForward(nn.Module):
    def __init__(self, n_embd):
        super().__init__()
        self.net = nn.Sequential(nn.Linear(n_embd, n_embd), nn.ReLU())

    def forward(self, x):
        return self.net(x)

FFN = feed-forward network; “feed-forward” means information only flows forward, never back (as opposed to a recurrent network, an RNN). It is the same structure as part 10’s hidden layer h = torch.tanh(emb.view(-1, 6) @ W1 + b1), with tanh swapped for ReLU = max(0, x). (Strictly it has one layer for now; section 8 makes it the standard two.)

The Linear combines, the ReLU sets a threshold

The FFN’s input is the four heads’ concatenated output. Try one detector on the two-head concatenation above: weights [0.1, 0, 0, 0.1], bias −1, looking for “head A’s dimension 0 is strong and head B’s dimension 1 is strong”:

       after the Linear                   after ReLU
h   10.00×0.1 + 0.00×0.1 − 1 =  0.00      0.00    ← head A strong, head B zero: only half there
o    2.69×0.1 + 1.46×0.1 − 1 = −0.59      0.00    ← neither is strong enough
u    8.87×0.1 + 5.30×0.1 − 1 =  0.42      0.42    ← both strong: it fires

It fires only on u = “it starts with h” and “this is u”. A Linear without the ReLU gives the h row a middling value and cannot say “only if both hold”; a ReLU without the Linear leaves each head on its own. Combine first, then test against the threshold.

One MLP, applied at every position

“Each position goes through a small MLP” more precisely means: the same MLP (one set of weights) is applied at each position separately — the positions do not get one each. nn.Linear acts only on the last dimension, so position 3’s output depends on position 3 alone: ffwd(x)[:, 3] == ffwd(x[:, 3]).

🔑 Karpathy: attention is “communication”, the FFN is “computation”. A Transformer block alternates the two. The standard FFN widens by 4 and narrows back — two Linears, 2 × d × 4d = 8d² parameters — while attention’s q, k, v and proj hold 4d². In real models the FFN holds about two thirds of the parameters, as experiment D’s parameter breakdown will confirm.

Parameters: 7553 + 32×32 + 32 = 8609.

Measured: 2.2495, but the top pick after “thou” is r

step 3000: train loss 2.2826, val loss 2.2922
step 4000: train loss 2.2432, val loss 2.2456
step 4999: train loss 2.2145, val loss 2.2495

val 2.2495, 0.24 below the bigram’s 2.49, in line with Karpathy’s roughly 2.24. Train and val begin to part for the first time (a gap of 0.035) — the first hint of memorising answers.

The question again:

The three rows finally differ — the last position really does see the earlier characters. But the top pick after “thou” is r, which never appears there in the text.

The reason is “ou”: across the text, r follows 27.1% of all ou (our, your, four, hour, honour). The model still leans on the local rule “the last two letters are ou” and hasn’t made full use of the th before them. Seeing is not the same as using.

Real words start to appear in the generated text — thy, to, your, for, of, though, four, now, our: the bigram forgot the first two letters by the time it reached the third, and now every position can look back 8 characters, enough for short words.

Step four: stacking 3 layers makes it worse

“Communication + computation” has happened once. To let the model think in several rounds, package it as a Block and stack them:

class Block(nn.Module):
    def __init__(self, n_embd, n_head):
        super().__init__()
        self.sa = MultiHeadAttention(n_head, n_embd // n_head)
        self.ffwd = FeedForward(n_embd)

    def forward(self, x):
        x = self.sa(x)       # communication
        x = self.ffwd(x)     # computation
        return x

self.blocks = nn.Sequential(*[Block(n_embd, n_head=4) for _ in range(3)])

The * unpacks the list: nn.Sequential(*[b1, b2, b3]) is nn.Sequential(b1, b2, b3).

Karpathy’s result: val does not improve, and may get worse. Three times the parameters, no gain. Two reasons:

• Backward: the gradient must cross every layer in turn, multiplied at each. With a local derivative of 0.3 per layer, three layers leave 0.3³ = 0.027 — the first layer receives 2.7%
• Forward: each layer’s output replaces its input entirely, so the original information has to be re-expressed at every layer — a game of telephone, garbled by the end

Part 12 said depth pays better but takes a whole toolkit to collect, and part 17’s troubleshooting table lists “add residual connections” as a remedy for vanishing gradients. Now the network really is deep, and it is time to collect.

Residuals: each branch only adds a little

The change is two characters:

x = x + self.sa(x)       # was x = self.sa(x)
x = x + self.ffwd(x)     # was x = self.ffwd(x)

A branch no longer replaces x; it adds an edit to x. Like a shared document: everyone reads the whole text and can append comments, but no one can delete the original.

★ Derivation: unrolling the recurrence

Each branch is a different function (with its own parameters), written f₁, f₂, …; each reads its own input:

x₁ = x₀ + f₁(x₀)
x₂ = x₁ + f₂(x₁) = x₀ + f₁(x₀) + f₂(x₁)
x₃ = x₂ + f₃(x₂) = x₀ + f₁(x₀) + f₂(x₁) + f₃(x₂)
⋮
xₙ = x₀ + f₁(x₀) + f₂(x₁) + … + fₙ(xₙ₋₁)

Two easy mistakes: the last term is fₙ(xₙ₋₁), not f(xₙ) — xₙ is the result and cannot appear in its own formula; and f₁, f₂, … are different functions, not one f applied n times (3 Blocks make 6 branches: each Block’s attention and FFN).

A numeric check: x₀ = [1.00, 2.00], f₁(x₀) = [0.10, −0.05] → x₁ = [1.10, 1.95]; f₂(x₁) = [0.02, 0.30] (reading x₁, not x₀) → x₂ = [1.12, 2.25] = x₀ + f₁(x₀) + f₂(x₁) ✅

The unrolled form says two things:

• x₀ stands alone at the front with coefficient 1 → the original information (which character, at which position) reaches the end untouched, and lm_head can see it directly
• later branches read inputs that contain every earlier branch’s result (f₃ reads x₂ = x₀ + f₁(x₀) + f₂(x₁)) → several rounds of communication: what round one gathered, round two gathers on top of

★ Derivation: the term that stays 1 on the way back

Differentiate one layer (I is the identity, Jₖ = ∂fₖ/∂xₖ₋₁ the branch’s own Jacobian):

∂xₖ / ∂xₖ₋₁ = I + Jₖ

The chain rule multiplies n of these; expand:

∂xₙ / ∂x₀ = (I + Jₙ)(I + Jₙ₋₁) ⋯ (I + J₁)
          = I  +  Σₖ Jₖ  +  Σⱼ<ₖ Jₖ Jⱼ  +  …  +  Jₙ ⋯ J₁

The first term, I, does not care how small the Jₖ are. Without residuals only the last term Jₙ ⋯ J₁ remains, and a long product of it vanishes. The scalar version shows the difference at a glance, local derivative 0.3, three layers:

without residuals: 0.3 × 0.3 × 0.3        = 0.027
with residuals:    (1 + 0.3)³ = 1 + 0.9 + 0.27 + 0.027
                                ↑ the path back along the trunk, always 1

On the computation graph this is part 4’s micrograd addition node: self.grad += out.grad, other.grad += out.grad — a (+) hands the gradient unchanged to both sides: one copy returns along the trunk undiminished, the other goes to the branch to update its parameters.

The idea comes from ResNet (He et al., 2015, image recognition): learning a difference is easier than learning the whole output. Suppose the best thing a layer can do is change nothing — without a residual, random weights must learn exactly the identity map, which is hard; with one, the branch only has to output 0, and shrinking weights toward 0 does that. “No change” is the default in a residual structure.

At initialisation, a deep net ≈ a shallow one

Freshly initialised Linear weights are small, so a branch’s output is usually much smaller than the x on the trunk: x = [1.00, 2.00], f(x) = [0.01, −0.02], x + f(x) ≈ x. Then x₁₂ = x₀ + (f₁ + … + f₁₂) ≈ x₀, and the whole model is roughly “lookup → lm_head” — part 24’s shallow model with nothing but an entry and an exit. Shallow models train easily: the simplest, bigram-like regularities come first, and as training grows the branch weights, f(x) starts to carry content — the branches “plug in” gradually.

Two accompanying changes

• A proj after the heads: self.proj = nn.Linear(head_size * num_heads, n_embd). Before a branch writes back to the residual stream, a Linear combines the heads’ side-by-side results — the [W_A W_B] from the derivation above. With it, heads × dims per head no longer has to equal n_embd
• The FFN widens 4× and narrows back: Linear(n_embd, 4·n_embd) → ReLU → Linear(4·n_embd, n_embd), the standard two-layer MLP. The second Linear is the projection that writes back to the residual stream (the original paper: d_model 512, inner layer 2048)

Karpathy’s val is about 2.08, and train starts to sit clearly below val — the bigger model can begin to memorise the training set.

LayerNorm: normalising along the other direction

★ Derivation: the residual stream’s variance keeps growing

The residual stream is a running sum: xₙ = x₀ + f₁ + f₂ + … + fₙ. Part 15’s rule: for independent variables, variances add; standard deviations do not. If every term has variance σ²:

Var(xₙ) = Var(x₀) + Var(f₁) + … + Var(fₙ) = (n + 1)·σ²
std(xₙ) = √(n + 1) · σ

Two signals with standard deviation 1 add to √2 ≈ 1.414 (V4 measured 1.413). Three Blocks add six times, about √7 ≈ 2.65×; six Blocks add twelve times, about √13 ≈ 3.61×. The deeper the net, the larger the scale, and later branches receive inputs on a completely different scale from earlier ones.

Why not BatchNorm

Parts 16 and 24 covered this; just the conclusions:

The fix: normalise each row

Part 24 said a matrix normalisation must first say which direction: BatchNorm goes by column (each feature, across all samples, x.mean(0)), LayerNorm by row (each position, across its own features, x.mean(-1)). LayerNorm looks only at one position’s own numbers, and all three problems above disappear.

one position's 4-dim vector (really 384 dims):
x = [2, 4, 6, 8]
mean = 5; variance = (9 + 1 + 1 + 9) / 4 = 5, std = 2.236
normalised = [−1.34, −0.45, 0.45, 1.34]    ← mean 0, variance 1
then × γ + β (learnable scale and shift, as in BN)

In the video Karpathy derives it from part 16’s hand-written BatchNorm: change mean(0) to mean(1) and delete the running-statistics lines. Real code uses nn.LayerNorm(n_embd), with γ 384 + β 384 = 768 parameters each.

Where it goes: Pre-Norm

def forward(self, x):
    x = x + self.sa(self.ln1(x))      # normalise, then attention
    x = x + self.ffwd(self.ln2(x))    # normalise, then FFN
    return x

LayerNorm sits at each branch’s entrance, not on the residual stream: every branch reads a normalised input, while the stream itself stays clean — the coefficient-1 straight path from the derivation above is untouched.

One more, self.ln_f (f = final), goes before lm_head: the residual stream has been added to many times, so it is normalised once on the way out. Stage two will meet RMSNorm — divide by the root mean square, no mean subtraction, cheaper — used by Llama and Qwen.

Measured: experiment C (3 layers + residuals + LayerNorm)

preset=small residual=True layernorm=True device=mps
0.0424 M parameters
step 0:    train loss 4.3103, val loss 4.3100
step 2000: train loss 2.1190, val loss 2.1600
step 4999: train loss 1.9747, val loss 2.0613

val 2.0613, matching Karpathy’s roughly 2.06. The 0.0424 M = 42,369 parameters match the itemised count exactly. The question, a third time:

Space replaces r at the top for “thou”, and g (though / thought) is second. That is several rounds of communication at work: section 7 communicated once and the last position mostly saw ou; now, with 3 layers and 6 branches, the th finally gets used. r hasn’t quite gone (12% after “thou”, against 0% in the text).

The train / val gap grows from section 7’s 0.035 to 0.087 — parameters went from 8609 to 42 thousand, and overfitting grows with them. The generated text already has a script’s shape: a character name, a colon, a newline, then lines — English at the level of phrases.

⚠️ This small model took 12.3 minutes on a Mac’s MPS. With a model this small, dispatching each tiny computation to the GPU costs more than the computation itself; evaluation isn’t cheap either (400 batches each time, 11 times — 4400 forward passes).

Scaling up + Dropout

Scaling up

Itemised (d = 384, per Block): attention 3d² + (d² + d) = 590,208, FFN (d·4d + 4d) + (4d·d + d) = 1,181,568, two LNs 2 × 768. Across the 6 Blocks and the rest:

The FFN holds about two thirds, as the 8d² : 4d² estimate predicted. And attention’s parameters do not depend on the context length: T from 8 to 256 adds not one. Part 12 ended by noting that every increment of the MLP’s block_size adds n_embd to the input width, so the approach does not scale — “this is exactly why the Transformer exists”. Here that promise is kept.

Where things stand: overfitting

10.79 million parameters against 1 million characters: the model can easily memorise the training set — train keeps falling while val stalls or rises. The gap was 0.035 in section 7 and 0.087 in experiment C; scaled up, it will be much worse.

The fix: Dropout

On every training forward pass, randomly zero part of the numbers and multiply the rest by 1/(1−p).

input:        [1.0, 2.0, 3.0, 4.0, 5.0]
random mask:  [ 1,   1,   0,   1,   1 ]      ← this time the third is picked (p = 0.2)
× 1.25:       [1.25, 2.5, 0,  5.0, 6.25]
next forward pass, a different set is zeroed

The model cannot rely on any fixed neuron or path, and has to learn redundant, robust regularities; you can also see it as training a random sub-network each step, ending up with an average of many.

Why multiply by 1/(1−p): each number survives with probability 1 − p, so the mask m is 1 with probability 1 − p and 0 with probability p:

E[ m · x / (1 − p) ] = (1 − p) · x / (1 − p) + p · 0 = x

In expectation, training sees the same values as without dropping. So evaluation simply switches Dropout off, and the layers after it receive values on the same scale as in training.

In GPT-2’s configuration ② and ③ share one setting, resid_pdrop (residual dropout); ① is attn_pdrop.

Dropout on the branch, never on the trunk

out = self.dropout(self.proj(out))    # ② before the multi-head output rejoins the stream

trunk x           = [1.00,  2.00, 0.50, −1.00]
edit f            = [0.40, −0.20, 0.10,  0.30]
after Dropout     = [0.50,  0.00, 0.125, 0.375]    ← the 2nd dropped, the rest ×1.25
x + f_dropped     = [1.50,  2.00, 0.625, −0.625]
                            ↑ not edited this time, but the original 2.00 is still there

Only part of the edit is dropped; not one number of the original x on the trunk moves. Written as x = self.dropout(x + ...) on the trunk, every layer would randomly erase original information and cut into the gradient’s straight path.

🔑 Dropout on the branch follows the same principle as LayerNorm at the branch entrance: keep the residual stream itself clean. Every change happens in a branch; the trunk only carries and accumulates.

model.eval() finally matters

Dropout is for training only; evaluation and generation must switch it off, or every prediction carries randomness. Part 23 left a line in estimate_loss(): model.eval() does nothing for the bigram, but write it now, because once Dropout arrives it must be off during evaluation — this is that moment. LayerNorm behaves the same in training and inference and needs no switch, so in a Transformer model.eval() is mostly about Dropout, unlike part 16, where it switched BN’s running statistics.

The complete picture

The model is now settled: embeddings → 6 Pre-Norm Blocks → ln_f → lm_head, a complete decoder-only Transformer — the GPT architecture. The figure below puts one forward pass, one Block, one head, the FFN, LayerNorm’s direction and the parameter breakdown in one place.

Experiment D: 1.4809

Time it first

Experiment D has 250× the parameters, 32× the context and 2× the batch. First a temporary 100-step speed test: about 1.06 seconds per step (evaluation included); taken apart, a pure training step is about 0.88 seconds → 5000 steps of training is about 73 minutes, plus 11 evaluations at about 21 minutes ≈ 1 hour 35. It actually took 93.1 minutes.

After those 100 steps val was 2.47 and the hou / you / thou rows were nearly identical — a 10.79-million- parameter deep network starts out behaving much like the bigram, exactly the “deep net ≈ shallow net at initialisation” starting point above.

Training

10.7889 M parameters
step 0:    train loss 4.2846, val loss 4.2820
step 1000: train loss 1.5327, val loss 1.7199
step 2000: train loss 1.3057, val loss 1.5512
step 3000: train loss 1.2012, val loss 1.5047
step 4000: train loss 1.1221, val loss 1.4877
step 4500: train loss 1.0857, val loss 1.4809   ← lowest val
step 4999: train loss 1.0503, val loss 1.4938   ← val starts rising
training finished, 93.1 minutes in total

This one step cut 0.58, more than every earlier improvement combined (0.43) — without changing a line of the model’s code, only its numbers. Once the structure is right, scaling up works.

The overfitting turning point

val bottoms out at step 4500 (1.4809) and then rises to 1.4938 while train keeps falling (1.0857 → 1.0503): the model has started memorising the training set. That matches the Shakespeare config shipped with nanoGPT, config/train_shakespeare_char.py:

eval_interval = 250 # keep frequent because we'll overfit
always_save_checkpoint = False   # save only when val improves

Karpathy knew this configuration would overfit, so evaluation runs every 250 steps (denser than the 500 here), and only the best-val version is kept. What was saved here is the final step (1.4938), not the best one. Not a large difference, but saving the best checkpoint is standard practice when training large models.

The fourth time: r is gone

Recomputed from the trained weights (gpt_large.pt) — under model.eval() there is no randomness, so the numbers come out the same every time:

Space 67.2% against the text’s 66.8%, g 20.7% against 16.7% — practically the text’s own distribution. The apostrophe in fifth place makes sense too: Shakespeare is full of thou’rt (thou art) and thou’lt (thou wilt). “you” matches the text closely as well (space 0.452, r 0.327, comma 0.104).

“hou” is further off (space 0.638, r 0.108, g 0.084, s 0.059; l, 15% in the text, misses the top five). The model isn’t at fault: in the text, “hou” nearly always has a letter before it (the s and w in should and would), while the test puts “hou” at the very start of the sequence, where the model can’t see any preceding letter. It can only guess what follows a bare “hou” opening a sequence, which is naturally not the statistic over every “hou” in the text.

The generated text reads like a stretch of a play:

QUEEN MARGARET:
What, are you?

GLOUCESTER:
You did, makes you upregant me.

KING HENRY VI:
The visitate Lords, or soul wo

The character names really exist in the plays (experiment C still invented them); sentence shapes, punctuation and archaic words all look right, but the sense doesn’t hold together. A 256-character context lets it keep track of which scene it is in and who is speaking.

Every step answers the same question

Multi-head attention, the FFN, residuals, LayerNorm, Dropout — each answers the same question: how to make better use of what came before. Residuals and LayerNorm are not goals in themselves; they are the precondition for going deep. Going deeper and bigger is what makes the model better at using context.

What has been written by now is, structurally, GPT-2. The next part compares it with the original paper to see which half it is, walks through nanoGPT to see where an industrial implementation differs, and looks at how ChatGPT grows out of a model like this.