3,481 Parameters: Same Numbers, Not Interchangeable

The previous part turned three characters into six numbers. The rest of the road sounds uneventful: the weighted sum was covered in part 1, cross-entropy in part 7.

But two things here are genuinely new, and they have the same shape — two ways of writing it produce identical numbers at costs an order of magnitude apart. The first is concatenating three vectors into six numbers; the second is computing the loss.

The whole chain, up front

emb    = C[X]                                    # (N, 3, 2)   previous part
h      = torch.tanh(emb.view(-1, 6) @ W1 + b1)   # (N, 100)    this part
logits = h @ W2 + b2                             # (N, 27)     this part
loss   = F.cross_entropy(logits, Y)              # scalar      this part

From here on, most of the work of reading the code is watching the shape.

The hidden layer is part 1’s formula

zⱼ = Σₖ wⱼₖ aₖ + bⱼ        aⱼ = σ(zⱼ)

h = torch.tanh(emb.view(-1, 6) @ W1 + b1) is the same thing. Term by term:

The same thing in three versions:

Part 5’s Neuron.__call__ was sum(wi*xi) + b followed by tanh — a for loop. This part’s @ replaces that for loop with a single library call. Not a word of the maths changed; what changed is how many it does at once.

A hundred at once, but each neuron computes its own share. The animation below steps through them one by one — note that the six inputs on the left never move; only the weights on the lines do:

The transpose convention: why W1 is (6, 100)

This is a notational trap, not a conceptual one. Part 1 writes wⱼₖ with j for output and k for input, giving a (out, in) matrix; here W1 is (in, out) = (6, 100), exactly reversed — because writing x @ W1 puts the samples on the left, forcing the shapes to be (N,6) @ (6,100).

Incidentally, PyTorch’s nn.Linear stores (out, in) internally and does x @ W.T on the forward pass, agreeing with part 1. Seeing (out, in) later does not mean you remembered it backwards — both conventions are in use, and how it multiplies on the forward pass tells you which.

Why it has to become a matrix multiplication

This part computes 228,146 samples × 100 neurons × 6 multiplications ≈ 137 million multiply-adds. A micrograd-style scalar for loop cannot carry that; matrix multiplication is the only route.

And matrix multiplication demands shapes that line up exactly. Hence:

want it fast ⟹ use matrix multiplication ⟹ must handle shapes ⟹ view / stride / contiguous

That is why the memory layout below gets so much space. It is not that PyTorch’s API is fiddly; it is the price of turning a per-neuron loop into a single matrix operation.

tanh is not decoration

Without a nonlinearity, (x @ W1 + b1) @ W2 + b2 expands to x @ (W1 @ W2) + (...), so two linear layers are equivalent to one and stacking more changes nothing. The nonlinearity is what makes “depth” mean anything.

Groundwork for V4. W1 = torch.randn(...) is a standard normal; six inputs each times a standard normal weight plus a bias gives a standard deviation of about 2.6 going into tanh — so roughly half the values in h end up pinned at ±1. tanh is nearly flat where it saturates, the gradient approaches 0, and those parameters stop learning.

How to check: (h.abs() > 0.9).float().mean(). This number returns in another form two sections down.

Why concatenate instead of sum

One sample’s emb is 3 rows of 2, and it has to become one row of 6:

"e" → [0.31, -0.82]
"m" → [1.05,  0.47]     ⟹  [0.31, -0.82, 1.05, 0.47, 1.05, 0.47]
"m" → [1.05,  0.47]

Not one value changed and not one was lost — only the shape was rearranged. Two reasons:

Why not compress with a sum or a mean? Because then abc and cba produce exactly the same result and the positional information is gone. Concatenation preserves order — rows 0 and 1 of W1 always belong to the first character, rows 2 and 3 to the second, rows 4 and 5 to the third. The model therefore knows which character sits where without being told separately.

This advantage disappears at the Transformer. Attention draws no distinction between orderings — to it the input is a set rather than a sequence, so positional information has to be added back deliberately as a positional encoding. This MLP does not need that, because concatenation wrote position into the coordinates.

A tensor is one strip of memory plus a manual

A PyTorch tensor = one contiguous, one-dimensional run of memory (the storage) + a manual for reading it (shape / stride / offset).

The same memory, four shapes

In memory, torch.arange(18) is 18 slots on a straight line: no rows, no columns, no rank. “Rank” is a property of the tensor object, not of the memory.

a = torch.arange(18)
a.view(18)        # rank 1
a.view(3, 6)      # rank 2
a.view(9, 2)      # rank 2
a.view(3, 3, 2)   # rank 3
# four tensors, identical data_ptr()

The one hard constraint: the dimensions must multiply to the number of slots. 3×6 = 18 ✓, 4×5 = 20 ✗.

address = Σ index[i] × stride[i]

The manual for b = a.view(3,6) reads “cut every 6 slots, into 3 segments”:

row 0 ← addresses 0..5      row 1 ← addresses 6..11      row 2 ← addresses 12..17

b[1][2] = skip 1 whole segment (6 slots each) + walk 2 slots inside it = address 8. There is no two-dimensional structure in memory; “two-dimensional indexing” only converts [1][2] into one address on a line.

stride[i] is the coefficient in that conversion: add 1 to the index along dimension i, and the address advances by that many slots. The row-major (C-order) derivation rule is “the last dimension varies fastest”:

stride[last] = 1
stride[i] = stride[i+1] × shape[i+1]      # working right to left

emb’s (6, 2, 1) reads plainly: moving to the next sample jumps 6 slots, moving to the next character within a sample jumps 2, moving to the next component within a character jumps 1.

view happens to be exactly the concatenation we want

emb is already laid out this way in memory:

sample 0: [0.31][-0.82][1.05][0.47][1.05][0.47] | sample 1: ...
           ← char 1 →   ← char 2 →  ← char 3 →

That run of memory is already “six numbers per sample, characters in order”. The (N,6) we want has stride (6,1) — moving to the next sample is still 6 slots, and the reading order has not changed at all.

emb.view(emb.shape[0], -1)     # free, only the manual changes
torch.cat([...], dim=1)        # same result, but allocates and copies

The most direct demonstration that view really moved no data is not comparing data_ptr() — it is changing a value:

a = torch.arange(18)
b = a.view(3, 6)
b[0, 0] = 999
print(a[0])        # 999 — a changed too

Both tensors point at the same storage. Forward passes rarely modify tensors in place, but when debugging by hand-editing a value, it may well have edited something elsewhere. .clone() is the real copy.

Not every concatenation is this lucky. Wanting “every sample’s first character grouped together” does not match the memory order, and then there is no choice but to copy. This one is free because the order we want happens to be the order we already have.

Why a transposed tensor cannot be viewed

Transposing also only swaps the manual; not one slot of memory moves:

b = a.view(3, 6)     # shape (3,6)  stride (6,1)
d = b.t()            # shape (6,3)  stride (1,6)     ← both fields swap

The definition of contiguous: read the elements in the tensor’s own logical order (last dimension varying fastest); if the addresses come out as 0,1,2,3,… without a break, it is contiguous.

d.view(18) asks for a one-dimensional tensor whose elements come in the order 0,6,12,1,7,…. But a one-dimensional tensor has only a single stride number s, and its address formula is i × s:

i=1 must land on address 6   → s = 6
i=2 must land on address 12  → s = 6   ✓
i=3 must land on address 1   → needs 3×6 = 1, no solution  ✗

An order that jumps around cannot be expressed as “advance a fixed number of slots each step”. So it raises:

view size is not compatible with input tensor's size and stride

“Non-contiguous” does not mean “gaps in memory”. d uses the same 18 slots, no more and no fewer. Non-contiguous means the reading order and the storage order disagree.

view / reshape / contiguous, and where -1 goes

Karpathy insists on view to keep the fact that this step is free visible. Production code uses reshape more often.

As for -1:

emb.view(228146, 6)            # ✗ both numbers hard-coded
emb.view(-1, 6)                # △ N is flexible, 6 is still hard-coded
emb.view(emb.shape[0], -1)     # ✓ neither is hard-coded

-1 is a placeholder meaning “work this one out yourself”: the unknown dimension = total elements ÷ the product of the others. A single view may contain at most one -1 — two unknowns with one equation has no solution.

Hard-coding breaks in two places. One is mini-batching, where N goes from 228,146 to 32. The other is tuning, where raising the embedding dimension from 2 to 10 should turn that 6 into 30.

The second is the nastier one — view(-1, 6) does not raise; it just produces the wrong shape (mixing numbers from different samples into one row) and blows up later at @ W1. Wrong at A, reported at B.

Parameters, inputs, activations: what gets updated

This is the easiest thing here to confuse.

How many weights does one neuron have? Six, not three. Three is “how many characters”, but a character is no longer a single number — it is a 2-dimensional vector, so the weight count is block_size × n_embd = 3 × 2 = 6.

Scaling up step by step:

(N, 6) @ (6, 100) = (N, 100)
       └─ the 6 is consumed ─┘

That middle 6 is the direction of the sum. What matrix multiplication consumes is always “features”; what it leaves are “samples” and “neurons”.

The parameter count has nothing to do with N

weights: 100 × 6 = 600
biases:  100 × 1 = 100
─────────────────────────
700 parameters

One sample means 700 parameters; a million samples still means 700. One set of parameters handles every sample — which is the precondition for a neural network working at all. h has N rows because it is a result, not a parameter.

Neuron count is not layer count

The hidden layer has 100 neurons, but it is 1 layer, not 100.

They are independent hyperparameters. For comparison: GPT-2 is 12 layers wide 768; GPT-3 is 96 layers wide 12,288. This part has only 2 layers — and the decade of progress since happened mostly in depth.

Part 5’s MLP(6, [100, 27]) is the most direct way to see it: two elements in the list means two layers.

C is a parameter too

parameters = [C, W1, b1, W2, b2]

This one is easy to miss: on the backward pass the gradient flows all the way back into C, adjusting where each character’s vector sits. The previous part said “what sits next to what is pushed there by gradients” — this is the path that pushes it.

Compared with micrograd, where the parameters were only weights and biases: this part adds a new kind of parameter — the representation of the input is itself being learned.

The output layer: neither dimension is free

Apply the template above: every layer’s weights are (previous width, this width), and its bias is (this width,).

The hidden layer’s 100 was chosen arbitrarily; the output layer must emit exactly 27 numbers, so neither dimension is free.

This layer has no tanh either: tanh squeezes values into (-1, 1), whereas logits are unnormalised log-probabilities and need to span arbitrary reals (+15 when very likely, −20 when very unlikely). Squashed to ±1 they cannot express that gap. Normalisation is softmax’s job.

Column k of W2 is the detector for character k

logitₖ = Σⱼ W2[j][k] × hⱼ + b2[k]        j from 0 to 99

The 100 numbers in W2[:, 1] are the detector for a, asking: how much does this particular combination of 100 features look like the letter a? W2[7][1] being a large positive number means “the stronger feature 7 is, the more this should output a”; a large negative number means the reverse.

b2[k] has a particularly clean interpretation: it is character k’s intrinsic preference — how common the character is on its own, before any context. In the extreme, if h were all zeros then logits = b2 and the model could only fall back on b2; training pushes it towards the natural frequency of the letters.

This is where the parameters are

That single layer accounts for 78%. Real language models start at vocabularies in the tens of thousands, so the output layer becomes the overwhelming parameter cost — which is why tied embeddings and hierarchical softmax were invented later.

The initial loss is not 3.30, it is 17

The sanity baseline is unchanged: guessing uniformly among 27 characters gives ln 27 = 3.2958.

But with randn initialisation the loss actually comes out around 17, more than five times worse than guessing. Working backwards: e⁻¹⁷ ≈ 4.1×10⁻⁸, so the probability the model assigns to the right answer is roughly one in twenty-four million. It is not guessing; it is confidently wrong.

Reconciling this with part 8. Part 8 said “the initial loss should be near 3.30, in practice slightly higher, around 3.5–4”, and here it is 17. Both are right; the difference is one extra layer.

Part 8 was a single layer: logits = onehot @ W is literally one row of W, 27 standard normal numbers with a standard deviation of 1.0. This part has two: the 100 numbers in h are each multiplied by a standard normal weight and summed, driving the standard deviation of the logits to about 9.

The loss is roughly “how far the largest logit leads the correct one”, and that gap scales linearly with the standard deviation — going from 1 to 9 takes the loss from 3.8 to somewhere near 18. This is not a bug, it is initialisation failing to control the scale, the same illness as the tanh saturation above.

Remember 3.2958. It is the yardstick for whether an initialisation is any good. V4 spends a whole episode on this, starting by multiplying W2 by 0.01.

Why F.cross_entropy is mandatory

The hand-written version was already written in parts 7 and 8:

counts = logits.exp()                              # (N,27) all positive
prob   = counts / counts.sum(1, keepdims=True)     # (N,27) each row normalised
loss   = -prob[torch.arange(N), Y].log().mean()    # scalar

None of those three lines is new; the line-by-line explanation is in part 7’s “where the other 26 terms went” and part 8’s “how the loss is computed”.

Switching to F.cross_entropy(logits, Y) is not about saving three lines — it is because those three lines will produce nan.

It is those six steps, plus a step 0

Step 0 is the only one you did not write, and it is precisely the one that keeps nan away. The other six are identical, which is why the numbers agree.

Worked by hand, scaled down to four classes. Input logits = [2.0, 1.0, 3.0, 0.0], correct answer Y = 2:

step 0  subtract max(3.0)  → [-1.0, -2.0, 0.0, -3.0]
step 1  exp               → [0.3679, 0.1353, 1.0000, 0.0498]
step 2  normalise         → [0.2369, 0.0871, 0.6439, 0.0321]   ← sums to 1
step 3  log(0.6439)       → -0.4402
step 4  Y=2, take index 2 → -0.4402   (the other three never participate)
step 5-6 negate, average  → 0.4402

F.cross_entropy(torch.tensor([[2., 1., 3., 0.]]), torch.tensor([2]))
# tensor(0.4402)   ✓ agrees

Three differences, only one of which changes the result

“Same numbers” and “interchangeable” are different claims. By analogy: sum(range(1, n+1)) and n*(n+1)//2 give the same result, but the latter is not “the same implementation” of the former.

Difference one: without the max subtraction, nan

logits = torch.tensor([-100., -3., 0., 100.])
prob = logits.exp() / logits.exp().sum()
print(prob)          # tensor([0., 0., 0., nan])   ← blown up

exp(-100) ≈ 3.7e-44      underflows to 0, harmless
exp(-3)   = 0.0498
exp(0)    = 1.0
exp(100)  = inf          ← overflow (float32 tops out near 3.4e38)
and then inf / inf = nan

The fix is softmax’s shift invariance: add the same constant c to every logit and both numerator and denominator pick up an e^c, which cancels, leaving the result unchanged.

e^(zᵢ+c) / Σⱼ e^(zⱼ+c)  =  (e^c · e^zᵢ) / (e^c · Σⱼ e^zⱼ)  =  e^zᵢ / Σⱼ e^zⱼ

So subtracting each row’s maximum is free: the largest becomes 0, exp(0) = 1, and everything else is below 1 — overflow becomes impossible.

logits - logits.max() = [-200, -103, -100, 0]
→ prob = [0, 1.4e-45, 3.8e-44, 1.0]        ✓ clean

Overflow and underflow are not symmetric. However far the negative side goes it merely becomes 0 and the answer is still right; one step too far on the positive side gives inf → nan and everything is lost. So only overflow needs defending against.

The threshold is ln(3.4e38) ≈ 88.7. And this part’s initial loss is 17 with logits already large, which training will pull further apart in both directions — the hand-written version really does blow up; this is not a theoretical risk.

Differences two and three: memory and speed

The hand-written version allocates a fresh (N,27) tensor at every step, and the backward pass needs them, so they cannot be released immediately:

At 27 classes this looks irrelevant. At a realistic vocabulary of fifty thousand, the same N is 45 GB — not an optimisation, but the difference between running and not running.

As for the backward pass: the hand-written version must walk back through five nodes, exp → sum → div → log → mean. But differentiating softmax and cross-entropy together simplifies into a single line:

∂L/∂logits = (prob − onehot(y)) / N

Using the worked example above (prob = [0.2369, 0.0871, 0.6439, 0.0321], Y = 2):

prob     = [0.2369, 0.0871,  0.6439, 0.0321]
onehot   = [0,      0,       1,      0     ]
──────────────────────────────────────────────
gradient = [0.2369, 0.0871, -0.3561, 0.0321]

It reads very plainly: the correct answer’s term has a negative gradient → push its logit up; every other term is positive, and the more probability the model gave it the harder it is pushed → push them down; when prob exactly equals the one-hot, every gradient is 0 → there is nothing left to learn.

PyTorch computes that single line and never walks back through the five steps. V5 is the episode that makes you derive this simplification yourself — a large part of why softmax plus cross-entropy is used everywhere is that its derivative comes out this clean.

Two traps in the signature

One: pass logits, not probabilities. F.cross_entropy applies softmax internally, so passing something you already softmaxed applies it twice. It does not raise; the loss simply refuses to come down, and it is very hard to track.

Two: Y is a class index, not a one-hot. Just give the index, torch.tensor([2]) — 26 of the 27 terms would be multiplied by 0 anyway.

That parameters list is an alarm

parameters = [C, W1, b1, W2, b2]
print(sum(p.nelement() for p in parameters))     # 3481

for p in parameters:
    p.requires_grad = True

It is not a PyTorch keyword, just an ordinary Python list. Because training does three things to every parameter:

for p in parameters: p.requires_grad = True    # once
for p in parameters: p.grad = None             # every round
for p in parameters: p.data += -lr * p.grad    # every round

Without the list, each of those is five lines. Real models have hundreds of parameter groups, and one left out simply never updates — silently.

That print is the alarm: change the embedding dimension and forget to change W1’s row count, and this number comes out wrong — caught before training even starts. The list holds references, not copies, so p.data += ... modifies W1 itself.

Compare the idiomatic PyTorch:

optimizer = torch.optim.SGD(parameters, lr=0.1)
optimizer.zero_grad()      # replaces the p.grad = None loop
loss.backward()
optimizer.step()           # replaces the p.data += loop

Karpathy writes the list by hand so that you can see there is no magic behind optimizer — it does those two for loops. Consistent with micrograd: part 5’s MLP.parameters() also returned a hand-assembled list of every Value, and this part only swaps the scalars for tensors.

One sentence for this part: 3,481 parameters, 2 layers, every shape lined up — and it has never been trained once. The initial loss is 17, five times worse than guessing.

The second half of V3 does four things: mini-batching brings a round’s 137 million multiply-adds down to a few tens of thousands (and flushes out that hard-coded 6 in view(-1, 6) along the way); the learning rate stops being guessed and gets swept for; the dataset is split into train / validation / test for the first time — part 8’s “the moment validation loss bottoms out and turns up is the moment to stop” finally becomes usable; and finally the tuning, plus plotting those 2-dimensional embeddings to see what they actually learned. The loss falling from 2.45 to 2.2 happens there.

As for that initial loss of 17, and the pile of values in h pinned at ±1 — they are one illness, and it waits for V4 and its episode on initialisation.