Maths Catch-Up · The Derivative of a Copy Is 1: How Gradient Reaches the Embedding Table

Parts 9 through 12 built the V3 MLP end to end, and C was treated throughout as “a matrix you look things up in”. But in the training loop it goes into parameters right alongside W1 and b1 — it is a parameter, so it must have a gradient.

The awkward part is that C sits in the middle of the chain, and the step it takes part in is not an operation at all; it is a copy:

emb = C[X]

There is a ready-made formula for differentiating a matrix product. There is none for differentiating “copy”. This part supplies it.

A gradient always has the shape of what it differentiates

In deep learning only one case ever comes up: the loss is a scalar, differentiated with respect to a vector or a matrix.

A gradient always has the same shape as the thing it differentiates. That is what lets a parameter update be a single line:

W -= lr * W.grad          # same shape on both sides, subtracted elementwise

This convention is called denominator layout. The other one is numerator layout, and the transposes land on opposite sides. When the same formula looks different in two textbooks, it is often just a difference of convention, not an error. Deep learning defaults to denominator layout, and so does everything below.

Two rules are enough

The derivative of a linear function is its coefficient

Take

zⱼ = Σᵢ xᵢ · W[i][j]

and expand the summation:

zⱼ = x₀·W[0][j] + x₁·W[1][j] + … + xₖ·W[k][j] + …
                                    ↑
                          only this term contains xₖ

Differentiating with respect to xₖ, every other term is a constant and vanishes; what remains is “constant × variable”, whose derivative is the constant itself:

∂zⱼ/∂xₖ = W[k][j]

Nothing to memorise. W[k][j] already means “how much xₖ influences zⱼ”, and that is exactly what the derivative asks.

The difference between linear and non-linear sits right here:

That second row is the root of tanh saturation: as h approaches ±1, 1 − h² → 0 and the gradient is choked off. The previous part’s closing line — “gradient loses half its size passing through tanh” — comes from this expression. Nothing about tanh is specially designed to do that; its derivative simply depends on the current value, and the current value has drifted into the saturated region.

That summation is the only reason anything is ever added

If L depends on u through several intermediate variables z₁, z₂, …:

∂L/∂u = Σⱼ (∂L/∂zⱼ) · (∂zⱼ/∂u)

That summation is the sole source of every “why do these get added” that follows. However many paths carry u’s influence to L, that many contributions get summed — no further explanation is required.

It is also what part 4’s _backward was about when it had to write += rather than =. Writing = throws the summation away, and whichever contribution is written last overwrites the rest.

The backward formulas of a linear layer

Written in matrix form, the two rules above give two expressions that hold for every linear layer:

z = xW   ⟹   ∂L/∂x = ∂L/∂z · Wᵀ        ∂L/∂W = xᵀ · ∂L/∂z

One thing, three names, one per field:

It is not a theorem of linear algebra; it is rule 1 plus rule 2, written as matrices.

VJP is a name worth knowing, because a framework never builds the Jacobian J explicitly. For a convolution or an attention layer, J is far too large to store. Computing only “J times a vector” sidesteps constructing the matrix at all. Every PyTorch operator’s backward is its own VJP.

What the transpose does. It changes no value; it only makes the dimensions line up:

(1,3) @ (6,3) has mismatched inner dimensions; (1,3) @ (3,6) lines up — so W gets transposed.

Mathematically that is reading one table in the other direction. In the forward pass z = xW, row k of W is “where xₖ flows”; in the backward pass ∂L/∂x = ∂L/∂z·Wᵀ, column j is “who gets a share of zⱼ’s gradient”.

The two formulas transpose different things for the same reason:

Nothing to memorise; go by shape. In the episode where he derives backpropagation by hand, Karpathy uses exactly this: write whichever arrangement lines up, then run a numeric gradient check on it.

The step at C: differentiating an index

Everything so far was ordinary matrix multiplication. C[X] is different because it is not an operation; it is a copy.

Two equivalent ways to write it

Form A (what the code says):

emb[n, t, d] = C[ X[n, t], d ]

Three subscripts locate a scalar: n is which sample, t is which character in the context (0/1/2), and d is which component of that character’s vector.

emb is (N, 3, 2) — three-dimensional, not 3×2. emb[3] is the 3×2 one. Each subscript removes a dimension; three of them leave a single number.

Form B (the one to differentiate):

emb = onehot(X) · C

Because a one-hot row has a single 1 in it, the matrix product degenerates into “take row number k”.

That is an ordinary x @ W — C is the first layer’s weight matrix. It isn’t written as a one-hot product only because 99% of that arithmetic would be multiplying by zero. So the real structure is three layers:

one-hot(27) ──@C──▶ (n_embd) ──@W1──▶ (n_hidden) ──@W2──▶ (27)

The Jacobian is a 0/1 selection matrix

The forward pass is a copy, and the derivative of y = x is always 1. So:

Neither condition can be dropped: the row has to match (row v of C must be the character this position actually used) and the column has to match (d = e; component 0 only affects component 0).

“Everything else” is easy to gloss over, so here it is spelled out. Take X = [2, 0, 2] and n_embd = 2, and fix one cell: when C[2][0] changes, what happens to each cell of emb?

So “everything else” splits three ways: row wrong / column wrong / both wrong. Each condition is asking its own question. X[n,t] = v asks whether this position used row v of C at all; d = e asks about the component — C[v][0] and C[v][1] are two independent numbers, and changing one does not touch the other.

Written out in full, the rows are the 6 cells of emb and the columns are the 6 cells of C:

Every row holds exactly one 1, marking where that cell was copied from. Read by column, it is the backward sorting rule:

Against an ordinary linear layer, the only difference is what the Jacobian looks like:

The derivative of “copied” is 1, the derivative of “unrelated” is 0 — there is nothing in between.

The 0s and 1s describe the derivative, not the values

This is the easiest thing to conflate: the values in emb are of course real numbers — they are exactly what C holds. The 0s and 1s describe a different object: the ratio by which emb moves when C moves.

In numbers. Let C[2] = [2.0, −1.0] and X = [2, 0, 2], so emb[0] = emb[2] = [2.0, −1.0]. Nudge C[2][0] from 2.0 to 2.01:

Look at the second row: the derivative is 0, but the value of emb[1][0] is 1.0, not 0. Two different things.

The derivative can only be 0 or 1 because the forward pass is an identity copy, emb[0][0] = C[2][0]. That is y = x, whose derivative is always 1 — whether x is 2.0 or −0.82 or 1000. For contrast: had the forward pass been emb[0][0] = 3.5 * C[2][0], the derivative would be 3.5. Indexing multiplies by no coefficient at all — or rather, its coefficient is identically 1.

So why isn’t C.grad made of 0s and 1s

Because the chain rule also multiplies in the upstream gradient. Using the numbers from the worked example below:

∂L/∂C[2][0] = 0.665 × 1 + (−0.755) × 1 = −0.090
              ‾‾‾‾‾   ‾         ‾‾‾‾‾‾   ‾
            upstream select    upstream select

The 0s and 1s are a switch, not a value. They decide only which upstream gradients get through and which are blocked; the ones that get through are still real numbers. So C.grad is full of numbers like −0.090 and 0.665 — it is just that some rows are entirely zero (every switch off) and some are several shares summed (a switch closed more than once).

Substituting the 0/1 back into the chain rule collapses most terms, and the sum over d disappears too (since d = e is required):

∂L/∂C[v,e] = Σₙ,ₜ  1[ X[n,t] = v ] · ∂L/∂emb[n,t,e]

In words: the gradient of row v of C is the sum of the emb gradients at every position in the batch that used character v.

Two properties: sparsity and accumulation

Sparsity — rows that went unused get a gradient of 0. A batch can only touch so many characters; rows of C it never touched have an all-zero gradient and do not move on that step.

That is why q had flown to the very top of the previous part’s scatter plot: it is too rare in names, most batches never touch row 17 of C at all, so that row has barely been updated and still sits near its random initialisation.

Accumulation — when the same character appears several times in one batch, the gradients add. This comes entirely from the summation in rule 2; nothing else is needed to explain it.

PyTorch does not actually build a one-hot matrix and multiply by it; it uses index_add_:

# conceptually equivalent to
C.grad = torch.zeros_like(C)
for n in range(N):
    for t in range(3):
        v = X[n, t]
        C.grad[v] += emb.grad[n, t]     # ← accumulate, not assign

This is what nn.Embedding does internally, and it supports sparse=True — which saves a great deal of memory once the vocabulary runs to tens of thousands.

Small enough to do by hand

Shrink everything until it fits on paper: a vocabulary of 3, n_embd = 2, one sample, no hidden layer.

C = [[1.0,  0.0],      # character 0
     [0.0,  1.0],      # character 1
     [2.0, -1.0]]      # character 2

X = [2, 0, 2]          # the context
Y = 1                  # the target

W = [[1,0,0],[0,1,0],[0,0,1],
     [1,0,0],[0,1,0],[0,0,1]]     # (6,3), chosen to be tidy by hand

Forward:

① look up   emb[0]=C[2]=[2,−1]   emb[1]=C[0]=[1,0]   emb[2]=C[2]=[2,−1]
② flatten   x = [2, −1, 1, 0, 2, −1]
③ logits    logit₀ = x₀+x₃ = 2    logit₁ = x₁+x₄ = 1    logit₂ = x₂+x₅ = 0
④ softmax   probs = [0.665, 0.245, 0.090]
⑤ loss      L = −ln(0.245) = 1.4076

Backward, starting from ∂L/∂logits = probs − onehot(Y) with Y = 1:

∂L/∂logits = [0.665−0, 0.245−1, 0.090−0] = [+0.665, −0.755, +0.090]

Those three add up to exactly 0 — the probabilities sum to 1 and so does the one-hot. A free self-check.

Through W: ∂L/∂xₖ = Σⱼ (∂L/∂logitⱼ) · W[k][j]. Each row of W holds a single 1, so every xₖ simply collects the gradient of the logit it points at:

∂L/∂x = [+0.665, −0.755, +0.090, +0.665, −0.755, +0.090]

Un-flatten, folding the 6 numbers back into (3,2):

Un-flattening and the forward view are the same operation, run in opposite directions. Not one value changes; only the shape is rearranged.

Down to C, grouped and summed according to X = [2, 0, 2]:

Why they add

Let u = C[2][0]. The forward pass copied it into two places: emb[0][0] = u and emb[2][0] = u. Apply rule 2:

∂L/∂u = (∂L/∂emb[0][0])·1 + (∂L/∂emb[2][0])·1
      = 0.665 + (−0.755)
      = −0.090

Both ∂emb/∂u equal 1, because both are identity copies. The entire reason they add is that summation.

The two shares have opposite signs and partly cancel, and that is only the arithmetic of a sum, nothing more.

Don’t anthropomorphise a gradient. “Position 0 wants it smaller, position 2 wants it larger” corresponds to no mathematical object, and it makes the sign easy to get backwards — when the gradient is positive, p -= lr * grad makes the parameter smaller. A gradient means one thing: raise u by ε and L changes by about (∂L/∂u)·ε. That is all.

Two pieces of verification code

The first runs every step of the hand calculation above:

import torch, torch.nn.functional as F

C = torch.tensor([[1.,0.],[0.,1.],[2.,-1.]], requires_grad=True)
W = torch.eye(3).repeat(2,1)          # (6,3)
X = torch.tensor([[2,0,2]])
Y = torch.tensor([1])

emb = C[X]
emb.retain_grad()                     # ← intermediates keep no grad by default
loss = F.cross_entropy(emb.view(1,-1) @ W, Y)
loss.backward()

print("loss    =", loss.item())               # 1.4076
print("dL/demb =\n", emb.grad[0])
print("dL/dC   =\n", C.grad)

# check that C[2] really is two shares summed
print("check:", (emb.grad[0,0] + emb.grad[0,2]).tolist(),
      "should equal", C.grad[2].tolist())

The last line prints two identical lists — direct proof that position 0 plus position 2 is C[2].

The second checks sparsity on the real model:

ix = torch.randint(0, Xtr.shape[0], (32,))
for p in parameters: p.grad = None
F.cross_entropy(forward(Xtr[ix]), Ytr[ix]).backward()

nz = (C.grad.abs().sum(1) > 0)
print("rows with gradient:", ''.join(itos[i] for i in range(27) if nz[i]))
print("characters in batch:", ''.join(itos[i] for i in sorted(set(Xtr[ix].flatten().tolist()))))

The two lines print the same thing — a direct check on that indicator function 1[X[n,t] = v].

Nothing new was introduced here. This part only works out what part 9’s “C is a parameter too” actually computes to: C is the first layer’s weight matrix, its Jacobian happens to degenerate into a 0/1 selection matrix, and so “differentiate with respect to the weights” degenerates into “group by row and sum”.

Back to the main line. The next episode fixes three things, all of them already measured: the initial loss should be 3.30 and is 17; far too large a fraction of h sits against ±1; and gradient loses half its size passing through tanh — that last expression being rule 1’s 1 − h² from above. Only once those are fixed does deepening the network make sense.