The Differences Are All Engineering: A From-Scratch GPT Beside the Paper and nanoGPT

Part 25 ended with a model that is, structurally, GPT-2. This part writes no new code; it does three things: compares it with the original paper to see which half it is; walks through nanoGPT to see where an industrial implementation differs; and looks at how ChatGPT grows out of a model like this.

The conclusion first: structurally there is almost no difference; the differences are all engineering.

Beside the paper: which half was written

The original 2017 Transformer is a translation model with two halves:

      encoder (left half)                decoder (right half)
self-attention (no mask)             self-attention (masked)       ← written
Add & Norm                          Add & Norm                     ← written
                  k, v ───────→     cross-attention (q from the decoder)  ← not written
                                    Add & Norm
feed-forward network FFN             feed-forward network FFN       ← written
Add & Norm                          Add & Norm                     ← written
   × N layers, reads Chinese           × N layers, emits English word by word

Part 25 wrote the right half minus cross-attention — what GPT calls decoder-only. The task is continuation: there is no source text to read in first and then consult while generating. Only when turning one kind of thing into another (Chinese → English, an image → a description) do you need an encoder plus cross-attention (part 24’s note ⑤).

The paper’s diagram says Add & Norm: add the residual, then normalise — Post-Norm, x = LN(x + f(x)). Part 25 wrote the Pre-Norm used from GPT-2 on: x = x + f(LN(x)). Reading the code against the paper’s diagram, the different LayerNorm placement is not a mistake; it is the later, improved form.

💡 Multimodal models (Qwen2.5-VL, say) have both halves: the vision part is an encoder (ViT, no mask) and the language part a decoder (masked). How the two connect — cross-attention, or splicing image tokens straight into the text sequence — is one of the central questions of multimodal design.

Merging the heads: c_attn, view, transpose

This is the biggest difference between nanoGPT and part 25, and the pattern you meet first when reading Llama or Qwen source (GQA is a modification of it).

Two ways to write it

Part 25: six Head objects, computed one after another, concatenated at the end.

self.heads = nn.ModuleList([Head(head_size) for _ in range(num_heads)])
out = torch.cat([h(x) for h in self.heads], dim=-1)

nanoGPT’s CausalSelfAttention.forward (the hand-written path, excerpted):

B, T, C = x.size()
q, k, v  = self.c_attn(x).split(self.n_embd, dim=2)                   # c_attn = Linear(C, 3C)
k = k.view(B, T, self.n_head, C // self.n_head).transpose(1, 2)   # (B, nh, T, hs)
q = q.view(B, T, self.n_head, C // self.n_head).transpose(1, 2)   # (B, nh, T, hs)
v = v.view(B, T, self.n_head, C // self.n_head).transpose(1, 2)   # (B, nh, T, hs)
att = (q @ k.transpose(-2, -1)) * (1.0 / math.sqrt(k.size(-1)))
att = att.masked_fill(self.bias[:,:,:T,:T] == 0, float('-inf'))
att = F.softmax(att, dim=-1)
att = self.attn_dropout(att)
y = att @ v                                                       # (B, nh, T, hs)
y = y.transpose(1, 2).contiguous().view(B, T, C)                  # = torch.cat
y = self.resid_dropout(self.c_proj(y))                            # = proj + Dropout

Mathematically identical (same multiply-adds, same result); the only difference is how many times work is sent to the GPU. Part 25 makes about 50 small calls per layer (6 heads × 3 Linears, plus each head’s attention); nanoGPT makes about 10 large ones. GPUs are best at a few large calls — one reason experiment D was slow on a Mac.

By hand: T = 2, C = 4, 2 heads, 2 dims each

The batch dimension is dropped.

① c_attn: its weight is just the heads’ small matrices side by side, W_attn = [Wq₀ | Wq₁ | Wk₀ | Wk₁ | Wv₀ | Wv₁] (4 × 12), in one matrix multiplication:

x @ W_attn =
t0: [ 2, 0,  0, 2,  1, 3,  1, 2,  3, 0,  1, 2 ]
t1: [ 2, 1,  1, 2,  0, 3,  3, 1,  1, 1,  2, 2 ]
      └q₀┘  └q₁┘  └k₀┘  └k₁┘  └v₀┘  └v₁┘

② split: cut every 4 columns → q, k, v, each (T, C) = (2, 4). The cut separates q/k/v, not heads — q = [[2,0 | 0,2], [2,1 | 1,2]]: the left two columns are head 0, the right two head 1, and each head’s q is already one complete block of columns.

③ view(T, 2, 2): not one number moves; each row of 4 is simply read as “2 heads × 2 dims”, giving a “position × head” table:

④ transpose(1, 2): swap that table’s rows and columns; each cell’s small vector moves as a whole, unchanged inside:

Head 0’s block [[2,0],[2,1]] is exactly q₀, and head 1’s [[0,2],[1,2]] is q₁ — identical to computing them separately. Likewise for k and v. In one index formula:

q_heads[b, h, t, d] = q[b, t, h·hs + d]

which is torch.stack(q.split(hs, dim=-1), dim=1): cut the column blocks apart and stack them.

⑤ Both heads’ attention in one go (scale 1/√2, lower-triangular mask):

head 0: att = [[1.00, 0   ],       y₀ = att @ v₀ = [[3.00, 0.00],
               [0.80, 0.20]]                         [2.61, 0.20]]
head 1: att = [[1.00, 0   ],       y₁ = att @ v₁ = [[1.00, 2.00],
               [0.50, 0.50]]                         [1.50, 2.00]]

(Head 0, row t1: q = [2,1] against k₀ = [1,3] and [0,3] gives dot products 5 and 3; divided by √2 and put through softmax, [0.80, 0.20]. Head 1’s two dot products are both 5, so half each.)

⑥ transpose back + contiguous + view:

[[3.00, 0.00, 1.00, 2.00],
 [2.61, 0.20, 1.50, 2.00]]   = torch.cat([y₀, y₁], dim=-1) ✅

★ Why “head” has to move forward

PyTorch’s batched matrix multiply works on the last two dimensions only; everything in front is treated as a stack of sheets, each computed on its own. The dimension order decides what is written on each sheet:

view can only split or merge dimensions, never reorder them (c_attn’s output is stored position after position), so transpose has to bring the head dimension forward.

Two easy misunderstandings

“After split, each head’s q is already together — why reshape at all?” The first half is right: the left two columns are head 0’s complete matrix. The trouble is that the heads’ column blocks sit side by side in one row, and a dot product adds up every column in the row:

t1's q · t0's k = [2, 1, 1, 2] · [1, 3, 1, 2] = (2 + 3) + (1 + 4) = 5 + 5 = 10
                                                  └ head 0 ┘  └ head 1 ┘

The two heads’ scores are summed into one — collapsing back to a single head. transpose is the pair of scissors that cuts the side-by-side blocks apart and stacks them.

“transpose(1, 2) transposes the q matrix” — it does not. It swaps the “position” and “head” dimensions; each head’s inner (T, hs) matrix is untouched. q, k and v remain three separate tensors, aligned by “sheet h = head h”, and the matrix multiply pairs them sheet by sheet.

What a “head” really is

A head is a group of dimensions that compute a dot product together and get their own softmax, giving one way of attending. The same 4 numbers: ungrouped (one head, scaled by 1/√4), t1’s allocation is [0.73, 0.27]; split into 2 groups, they give head 0’s [0.80, 0.20] and head 1’s [0.50, 0.50]. view decides the grouping; training makes each group learn a different pattern.

In the small example T, the head count and hs all happen to be 2, so the shapes before and after transpose look the same. With T = 3 it is clear:

Why .contiguous()

transpose changes only how the data is read (the stride), not the data (part 10 covered view and stride). The y above (head, T, hs) sits in memory as [3, 0, 2.61, 0.20 | 1, 2, 1.50, 2] (head 0, then head 1); view(T, 4) needs [3, 0, 1, 2 | 2.61, 0.20, 1.50, 2] (t0, then t1). So .contiguous() first copies it in the new order; calling view without it raises an error — or use .reshape(), which copies when it has to.

Check: both ways give the same result

Random weights for B = 2, T = 5, C = 12 and 3 heads, computed both ways:

ours = torch.cat([one_head(i) for i in range(nh)], dim=-1)
W_attn = torch.cat(Wq + Wk + Wv, dim=1)                # the heads' small matrices side by side → (C, 3C)
q, k, v = (x @ W_attn).split(C, dim=2)
q, k, v = [t.view(B, T, nh, hs).transpose(1, 2) for t in (q, k, v)]
...
torch.allclose(ours, y.transpose(1, 2).contiguous().view(B, T, C), atol=1e-6)   # True
y.transpose(1, 2).view(B, T, C)
# RuntimeError: view size is not compatible with input tensor's size and stride ...

Threads picked up

• Part 24 said k.transpose(-2, -1) uses negative indices because k becomes 4-D with several heads — here it is: with (B, nh, T, hs), only the last two dimensions swap
• Part 25 said GPT-2’s configuration calls its two Dropouts attn_pdrop / resid_pdrop — attn_dropout is Dropout ①, resid_dropout is Dropout ②
• Heads don’t communicate with each other, for the same reason as part 24’s note ③ (no communication across the batch); heads exchange information in the c_proj that follows (the [W_A W_B] from part 25’s concatenation derivation)

One sample is T questions, and parallel equals serial

During training idx is a (B, T) table of integers, not (B, T, 384): B = 64 unrelated chunks of text cut at random, T = 256 characters each. Only after the embedding lookup does each id become 384 numbers. B and T never change inside the model; every computation works on the last dimension: id → 384-dim representation → 65 scores.

What goes in is known text, and a chunk of T characters is T questions:

The answers y are x shifted one place right, all already in the text; “unknown” only holds for the model, enforced by the mask hiding the right-hand side. The T losses average into one → one backward pass → one parameter update.

Serial vs parallel: serially, every prefix is fed separately, and across T forward passes the early positions are recomputed again and again — T(T+1)/2 positions in all, about 128 times as many at T = 256. A Transformer produces all T positions in one forward pass, and position t’s result is exactly what feeding the first t+1 characters alone would give: embeddings, LayerNorm and the FFN all work position by position, and attention is masked from the right. Checked with experiment D’s trained weights:

idx = torch.tensor([encode('First Citizen:')])     # (1, 14)
full = model(idx)                                   # parallel: one pass, all 14 positions
for t in range(14):
    part = model(idx[:, :t+1])                      # serial: only the first t+1 characters
    torch.allclose(full[0, t], part[0, -1], atol=1e-5)   # True for all 14

This is the key to the Transformer displacing the RNN: an RNN must compute in order and cannot parallelise over T in training. Conversely, at generation time the next character really is unknown, so it has to be serial; part 25’s generate() refeeds the whole sequence at every step, which is serial recomputation — real inference uses a KV cache to keep earlier positions’ k and v and compute only the new position. That works precisely because earlier positions are unaffected by later ones.

Walking through nanoGPT’s model.py

karpathy/nanoGPT’s model.py is about 330 lines, read in four rounds:

What is new in the source falls into three tiers:

Round 1: the parts

• A home-made LayerNorm: only so that β can be switched off (at the time nn.LayerNorm had no bias=False), with the computation left to F.layer_norm. weight = γ starts at 1, bias = β at 0, like bngain / bnbias in part 16’s hand-written BN
• Flash Attention: one line, F.scaled_dot_product_attention(q, k, v, ..., is_causal=True), replaces the five hand-written ones, with is_causal in place of the mask. It never stores the full (B, head, T, T) matrix (in experiment D, 64 × 6 × 256 × 256 ≈ 25 million numbers per layer), saving memory and time — more so the longer T gets
• MLP = FeedForward: c_fc (C→4C) → GELU (a smoother ReLU that squeezes negatives to small negative values instead of cutting them to 0) → c_proj (4C→C) → dropout. The c_ prefix comes from the Conv1D in OpenAI’s original code
• The Block, word for word: x = x + self.attn(self.ln_1(x)), x = x + self.mlp(self.ln_2(x)) — only the names differ

★ A correction: a bias on q is not wasted

GPT-2’s c_attn has biases. Part 24 said a bias “adds the same constant to every score in a row, so it does nothing” — only half right (that sentence has been fixed). Expand a biased score:

(xₜW_q + b_q) · (xₛW_k + b_k) = xₜW_q · xₛW_k  +  xₜW_q · b_k  +  b_q · xₛW_k  +  b_q · b_k

Take row t (position t looking at every s); softmax runs over s:

A bias on q adds, for each position s being looked at, a score for how much s deserves attention regardless of who is looking. A check with random weights: add only a k bias and the softmaxed row is unchanged; add only a q bias and it changes. (A bias on v amounts to adding a fixed vector to the output, which the next proj’s bias can absorb.) The video’s bias=False is a simplification, and nanoGPT’s training script switches biases off by default too (its comment: slightly better and faster) — switching them off is fine, but not because they are useless.

Round 2: assembly

• @dataclass GPTConfig: a Python data class whose defaults are GPT-2 small (12 layers, 12 heads, 768 dims, 1024 context). Experiment D is GPTConfig(block_size=256, vocab_size=65, n_layer=6, n_head=6, n_embd=384, dropout=0.2). GPT-2’s vocabulary of 50257 is padded to 50304 = 64 × 786, because GPUs are faster on multiples of 64

• ModuleDict gives parameter names a transformer. prefix (transformer.h.0.attn.c_attn.weight), matching the names in GPT-2’s weight files on Hugging Face exactly — preparation for loading weights. There is also a fourth Dropout: drop(tok_emb + pos_emb) (GPT-2’s embd_pdrop)

• Weight tying, self.transformer.wte.weight = self.lm_head.weight: the token embedding table is (vocab, C), and lm_head’s weight, stored the nn.Linear way as (out, in), is also (vocab, C) — same shape, so both names point at one tensor. It is used twice, so on the way back the two gradients add (part 4’s +=). The intuition: logits[j] = x · e_j — the more x resembles a token’s embedding, the higher that token scores. GPT-2 saves 50257 × 768 ≈ 38.6 million parameters this way (about 31%); experiment D’s tiny vocabulary saves only 24,960

• GPT-2 initialisation: every Linear and Embedding is N(0, 0.02), biases 0; weights whose names end in c_proj.weight (the two projections that write back to the residual stream) shrink further to 0.02 / √(2 · n_layer). The reason is part 25’s derivation: the residual stream is added to 2 · n_layer times, and independent signals add variances,

  Var(x_L) ≈ Var(x₀) + 2·n_layer · σ²        shrink each branch by √(2·n_layer) → the total added is back to order σ²

  stopping the residual stream from swelling at the source; with experiment D’s settings that is 0.02 / √12 ≈ 0.0058. Compared with PyTorch’s defaults: a 768-wide Linear is similar (about 0.021), but an Embedding defaults to N(0, 1), 50 times larger — part 25’s model trained well thanks to LayerNorm catching it

• get_num_params subtracts the position embeddings by default (the convention for reporting model size) and counts tied parameters once. nanoGPT’s version of experiment D has 10,745,088 in all, printed as 10.65M

• Four differences in forward: ① an explicit assert that the length is within block_size; ② view(-1, V) flattens only when computing the loss; ③ ignore_index=-1: positions whose answer is −1 add no loss — exactly what fine-tuning on the answer alone relies on later (Hugging Face uses −100); ④ at inference only the last position is computed, self.lm_head(x[:, [-1], :]), skipping lm_head at T−1 positions, with the list [-1] keeping the time dimension

Round 3: generation

logits = logits[:, -1, :] / temperature
if top_k is not None:
    v, _ = torch.topk(logits, min(top_k, logits.size(-1)))
    logits[logits < v[:, [-1]]] = -float('Inf')

Temperature: divide the logits by T, then softmax. Take logits = [3.0, 1.8, 0.6, −0.3, −1.0] (space, g, comma, s, r — at T = 1 close to what experiment D gives after “thou”):

T < 1 widens the gaps and plays safe; T > 1 is more random; T → 0 is argmax, T → ∞ picks at random. The temperature in the ChatGPT API is this one line (nanoGPT’s sample.py defaults to 0.8). Part 24’s ÷ √dₖ in note ⑥ is, at heart, setting a temperature for attention’s softmax too.

top-k: keep the k highest scores and set the rest to −∞ (the same trick as the mask). With k = 2 the truncated softmax is [0.77, 0.23, 0, 0, 0]. v[:, [-1]] takes the k-th largest value, the list index keeping (B, 1) for broadcasting. It cuts off the long tail: in a 50-thousand-token vocabulary, tens of thousands of unlikely tokens can add up to a few percent, and drawing one gives garbage; a 65-character model barely needs it. Chat models more often use top-p.

crop_block_size: shortening the context changes only two things — the position embedding table (keep the first block_size rows) and the mask (take the top-left corner). Every other parameter stays put, concrete evidence for part 25’s “attention’s parameters don’t depend on the context length”. The flip side: learned position embeddings can shrink but never grow (a position never seen has no row) → hence RoPE, coming later, which looks nothing up and rotates q and k by a formula; Llama and Qwen both use it.

Round 4: engineering

from_pretrained: build an empty model of the same structure → take both state_dicts → drop the masks (buffers are in state_dict too, as part 25 noted) → assert the counts match → copy by name. Vocabulary 50257, context 1024 and biases are fixed (the configuration must match the weights); only dropout may change.

Twelve times the model, 64 per head throughout — direct evidence for part 25’s “bigger models get more heads”.

Four kinds of weight are transposed when copied (c_attn, the two c_proj, c_fc): OpenAI’s original code uses Conv1D, storing weights as (in, out) and computing x @ W; nn.Linear stores (out, in) and computes x @ W.T. The copies run inside torch.no_grad(): modifying parameters in place needs gradients off. Hugging Face’s AutoModelForCausalLM.from_pretrained(…) is the ready-made version of this procedure, and its size mismatch / missing keys errors are these asserts.

configure_optimizers: weight decay pulls weights a little toward 0 each step (w ← w − lr·grad − lr·wd·w), a form of regularisation. Parameters with p.dim() >= 2 (matrices, embedding tables) decay; biases and LayerNorm’s γ, β don’t — pulling γ toward 0 would weaken the signal. Experiment D used AdamW’s default 0.01 on every parameter alike; nanoGPT’s train.py defaults to 0.1 on matrices only — one reason for the gap between experiment D (1.4809) and nanoGPT (1.4697), the other being the learning-rate schedule. Filtering out parameters with requires_grad=False is the key to LoRA’s memory savings: billions of frozen parameters stay out of the optimiser, and AdamW keeps no momentum or variance for them.

★ estimate_mfu: the 6N estimate

MFU = actual compute per second ÷ the hardware’s peak. The compute per token:

forward, per parameter: one multiply + one add                 → 2 operations
backward: gradients for inputs and for weights, about 2× forward → 4 operations
total                                                           → 6 per parameter, 6N for the model

plus q @ kᵀ and att @ v (no parameters, proportional to T): 12 · L · H · Q · T

(N excludes the position embeddings; L, H and Q are the layer count, head count and dims per head.) For experiment D:

Each character seen about 82 times is the root cause of part 25’s overfitting. Real pre-training usually sees each piece of data once or twice, and overfitting is rarely an issue; fine-tuning, with little data seen several times, is where it needs watching again. Before renting GPUs later, “compute per token × total tokens ÷ throughput” estimates the time and cost.

From GPT to ChatGPT

A model that has only been pre-trained can only continue text. Ask it “What is the attention mechanism?” and it may carry on with “What is a residual connection? What is LayerNorm? …” — treating the question as the start of a list of questions, perfectly reasonable for something that continues documents.

Structurally there is almost no difference — GPT-3 is a bigger gpt.py, and nearly all the difference is scale (experiment D’s “same code, bigger numbers, big improvement”, writ large). From continuation to conversation, OpenAI’s published approach has three steps:

1. Supervised fine-tuning (SFT): tens of thousands of hand-written “question → good answer” pairs, still trained by predicting the next token — only the data changes; it is the same training loop as part 25
2. A reward model: generate several answers to one question, have people rank them, and train a model that scores answers
3. Reinforcement learning (RLHF): keep adjusting the language model by the reward model’s scores — now optimising “do people like the whole answer”, no longer how well the next token is guessed

Pre-training takes the bulk of the compute; fine-tuning uses far less data and compute but decides whether the model is useful. Simpler alternatives such as DPO now exist. Later in this series comes step 1: Qwen has already been pre-trained and aligned by Alibaba, and the plan is supervised fine-tuning on top of it — no pre-training of one’s own, but an understanding of every piece inside, which is what the road from V1 to here was for.

Wrapping up V7

Karpathy’s episode ends here. From a hand-written autograd engine (part 4) to a model structurally equivalent to GPT-2:

Having compared it with nanoGPT, the conclusion holds: the from-scratch model and GPT-2 barely differ in structure; the differences are all engineering — heads merged into one computation, Flash Attention, weight tying, initialisation, grouped optimiser settings.

What comes next is not modern architecture but the tokenizer (episode eight) — the 65-character vocabulary in this code was defined by hand, whereas a real model’s is derived from a corpus, and what gets counted there is bytes rather than characters. After that comes reproducing GPT-2 (124M) on real web data.

RMSNorm, RoPE, SwiGLU and GQA wait until after that: they are changes made to the 124M code, measured against a miniature Qwen3, not something that picks up directly from this Shakespeare model. The swaps themselves are not hard — RMSNorm simplifies LayerNorm, RoPE replaces the position embedding table, and GQA changes one thing in those three lines above that merge the heads.