The Line That Looks Right: 96 Lookups Landing on 27 Rows, and the End of Ex1

The previous part’s thirteen steps took 1883 parameters, leaving W1, b1 and C — 2254 in all.

This part finishes in five steps, with no forks — from hprebn all the way back to the embedding table C, five steps in a straight line. The first three are a copy of the previous part; the last two each hold something that has not appeared before.

The three forward lines:

emb    = C[Xb]                          # (32,3,10)  lookup
embcat = emb.view(emb.shape[0], -1)     # (32,30)    flattened, rearranged not computed
hprebn = embcat @ W1 + b1               # (32,64)    Linear 1

❗ Xb is (32,3), not one-dimensional. That differs from the Yb (32,) part 19’s step ① used — C[Xb] fetches 96 things at once (32×3), while logprobs[range(n), Yb] fetches only 32. This one difference is the source of all of step ⑤’s trouble.

A lookup turns integers into vectors, so the rank goes up by one:

Xb      (32, 3)          32 samples × 3 character indices
   ↓ C[Xb]  each integer replaced by a 10-dimensional vector
emb     (32, 3, 10)      the extra 10 is n_embd

The first three steps: only the names change

The forward hprebn = embcat @ W1 + b1 has the same structure as the previous part’s logits = h @ W2 + b2:

dembcat = dhprebn @ W1.T        # (32,64) @ (64,30) → (32,30)
dW1 = embcat.T @ dhprebn        # (30,32) @ (32,64) → (30,64)
db1 = dhprebn.sum(0)            # (32,64) → (64,)

Take the previous part’s three lines, swap h for embcat and W2 for W1, and these are what come out — no thought required. db1 is likewise sum(0) without keepdim, since b1 is shaped (64,) and is already one-dimensional.

Step four: put the shape back

demb = dembcat.view(emb.shape)      # (32,30) → (32,3,10)

❗ First, the forward view did not modify emb

“Didn’t the forward already flatten it — why does the backward have to put it back?” — what got flattened was embcat, not emb. view returns a new view; emb itself is still (32,3,10).

The iron rule part 13 laid down earns its keep here: every forward variable has a gradient of the same shape.

Both variables exist, so both gradients have to, each following its own shape.

Every local derivative is 1

The forward moves things rather than computing them — each output position’s value simply is some input position’s value:

embcat[k][10j + d] = emb[k][j][d]

so:

∂ embcat[k][10j+d] / ∂ emb[k][j][d] = 1

One to one, coefficient 1, everything else zero. This is the extreme case of shape A — not even a multiplication is needed:

dembcat[0]  (9,)                          demb[0]  (3,3)
[g0, g1, g2, g3, g4, g5, g6, g7, g8]  →    [g0, g1, g2]
                                            [g3, g4, g5]
                                            [g6, g7, g8]

Nine numbers, not one of them changed, only regrouped.

💡 Pass emb.shape rather than writing view(32,3,10) by hand — then a different batch size or block size cannot break it. Same reason part 19’s step ① used zeros_like instead of torch.zeros(32, 27).

🔑 The backward of a shape operation

🔑 One test decides all of it: did the operation change the count of numbers. view has 960 numbers on both sides (32×3×10 = 32×30), so the count is unchanged ⟹ one to one ⟹ the backward is simply its own inverse. Broadcasting and summing do change the count, which is why one has to sum to gather and the other has to copy to send.

This table also swallows shapes B and C from the previous two parts: a broadcast’s backward being a sum and a sum’s backward being a broadcast were always just two special cases of this one test.

Part 10 covered why view is free (it shares the storage and only swaps in a different set of instructions) — that was the forward side; this is the backward side.

Why this step has to exist

The steps on either side want different shapes: matrix multiplication only eats two dimensions, so the forward has to flatten; a lookup is three-dimensional by nature, so the backward has to restore.

Without restoring, step ⑤ would have to work out column ranges itself:

dC[ix] += dembcat[k, j*10 : j*10+10]     # ❌ offsets by hand, easy to get wrong
dC[ix] += demb[k, j]                     # ✅ the indices come apart on their own

Step five: scatter and accumulate

The same row of C may have been fetched many times. 32×3 = 96 lookups, and C has only 27 rows — 3.6 per row on average, and a dozen or more for common letters like a and e.

Xb = [[0, 5, 13],      sample 0 took C[0], C[5], C[13]
      [5, 13, 13],     sample 1 took C[5], C[13], C[13]   ← two hits on C[13] within one sample
      ...]

Mathematically:

∂L/∂C[c][d] = Σ  ∂L/∂emb[b][k][d]        summed over every (b,k) with Xb[b][k] = c

The gradient of C’s row c is the sum of the gradients at every position that fetched row c.

dC = torch.zeros_like(C)          # lay down zeros: the accumulator's initial value
for k in range(Xb.shape[0]):      # k runs over the 32 samples
    for j in range(Xb.shape[1]):  # j runs over the 3 positions
        ix = Xb[k, j]             # which row of C this position originally fetched
        dC[ix] += demb[k, j]      # add the gradient back into that row

💡 demb[k, j] is a 10-dimensional vector, not a single number. demb is three-dimensional (32,3,10), and giving it two integer indices drops two ranks, leaving only the last. dC[ix] is (10,) as well, which is what makes the += work.

❗ Why it has to be +=

With an assignment, dC[13] ends up holding only demb[1][2] — the first two contributions were overwritten.

The accumulation rule — “a variable used N times gets its gradient added N times” — has come up twice already (part 19’s counts, the previous part’s bndiff), but in both N was 2, and visible in the code.

Here N varies — it depends on how often each letter occurs in this batch, and is only known at run time. This is the one place in Ex1 where the number of paths cannot be counted by reading.

⚠️ The line that looks right

dC[Xb] += demb        # ❌ looks entirely correct; repeated rows are added only once

That line reads far cleaner than the double loop, and it does not error. But it is wrong: with numpy and PyTorch fancy indexing, an in-place add takes effect only once at a repeated position.

The reason is that dC[Xb] += demb decomposes into read, add, write back rather than an atomic accumulation — when two positions in Xb both point at row 13, both write-backs carry a result computed from the same old value, and the second overwrites the first.

There are two correct vectorised forms:

dC.index_add_(0, Xb.view(-1), demb.view(-1, 10))              # built for repeated accumulation
dC = F.one_hot(Xb.view(-1), 27).float().T @ demb.view(-1,10)  # or turn it into a matmul with one_hot

Karpathy uses the double loop for teaching clarity — 96 iterations, each of them obvious.

🔑 This is the series’ first trap that neither errors nor is obvious. Every trap hit before this one either raised an exception or was caught by cmp: a wrong sum axis reports a shape mismatch, a misplaced transpose fails on the inner dimension, a missing .clone() turns cmp red on the spot. This line trips none of the three — the shape is right, nothing errors, and most cells even hold the correct value; only the rows that were fetched more than once come out too small. The approximate column of cmp is the only thing that will tell on it.

zeros_like is not ones_like

dC = torch.zeros_like(C)

Zero is the initial value, for two reasons: an accumulator has to start clean, and a letter absent from this batch should have a gradient of 0 anyway.

print((dC.abs().sum(1) == 0).sum().item())    # how many rows are all zero

Those rows are simply not updated by this step — which is normal, and part 13 did that accounting (a row only has a gradient if it was used).

⚠️ Do not confuse it with ones_like: zeros_like is an accumulator’s initial value (an empty container), while ones_like is a table of local derivatives (every position’s derivative being 1, used in shape C’s broadcast). The two functions look alike and do completely different jobs — the previous part’s steps ⑥ and ⑨ used the latter.

Set against part 19’s step ①

Both are “the forward picked a subset, the backward puts it back”, but one point differs:

🔑 Part 13’s appendix recorded that “the embedding table’s gradient must use +=, and dropping the summation sign makes it wrong” — that was a conclusion read off a Jacobian; this time it is an implementation written by hand.

The end of Ex1

From loss all the way back to every single parameter, twenty-odd atomic operations, not one skipped.

The five backward patterns, plus this part’s two rules, are the whole of it:

This whole stretch of backward() does exactly these things and nothing more. From the next section on it goes back to being one line of loss.backward() — except that this time you know what is inside that line.