820× Faster: A Noisy Gradient and a Curve That Only Rules Out

The previous part finished the network: 3,481 parameters, 2 layers, every shape lined up — and it had never been trained once.

This part gets it running. What is interesting is that getting it running takes two deliberate imprecisions: first a “wrong” gradient computed from 32 samples, then a coarse sweep to pick the learning rate. The first pays off cleanly; the second comes with a condition — and that condition is what this part is really about.

A step does not count samples

This is the easiest thing here to confuse. The 100 in for step in range(100) is not 100 samples.

emb = C[X]        # X is all 228,146 samples

One step consumes all 228,146 samples. A hundred iterations means the dataset was seen a hundred times over — 22.8 million sample-passes.

A step counts how many times the parameters were updated, not how many samples went through. One execution of p.data += -0.1 * p.grad is one step.

Three terms worth separating:

Steps and epochs coinciding under full-batch training is a coincidence. After mini-batching, one epoch takes 228,146÷32 ≈ 7,130 steps, and the two part ways. A paper saying “100k steps” and one saying “3 epochs” are describing completely different things.

Why not update once per sample

Because there is only one set of parameters, and the samples have to decide together.

Sample A says W1[3][7] should go up, sample B says it should go down — who wins? The right move is to average every opinion and then move once. That is precisely what the .mean() inside F.cross_entropy does; it was step 5 of the hand-written version in the previous part.

Getting it running: 100 full-batch steps

The three-part structure of the training loop was covered in part 6, compared side by side in part 8, and printed once more in the previous part’s parameters section, so it is not repeated here. Straight to the numbers:

The fall per 20 steps almost halves each time. The starting point is on an extremely steep slope (randn initialisation makes the logits absurdly large), so the first steps have enormous gradients; the closer to the floor, the gentler the slope and the slower the going.

8.2 seconds for 100 steps = 82 milliseconds per step. Reaching bigram’s 2.45 would take an estimated 30–50k steps ≈ 68 minutes. And the next two sections need experiments run over and over — this loop simply cannot be used for tuning.

The p.grad is None trap

TypeError: unsupported operand type(s) for *: 'float' and 'NoneType'

The cause is that requires_grad was never switched on. Tensors from torch.randn do not track gradients by default; re-running the cell that builds the parameters without re-running requires_grad = True breaks the chain.

The awkward part is that backward() does not complain — it simply does nothing. p.grad stays None, and the explosion waits until -0.1 * None. Once again, wrong at A, reported at B — the same shape of bug as the previous part’s view(-1, 6).

One line diagnoses it:

for name, p in zip(['C','W1','b1','W2','b2'], parameters):
    print(name, p.requires_grad, p.grad is None)

The test: backward() has been called and p.grad is still None → it is a requires_grad problem.

The prevention: keep “build the parameters” and “switch on gradients” in the same cell so they always run together. This is the single most common class of Jupyter bug — the order cells run in has nothing to do with the order they appear on the page.

Overfitting a tiny slice: the most general debugging move there is

Train on the first 5 samples over and over and see whether the loss can reach nearly 0.

Xs, Ys = X[:5], Y[:5]

for step in range(200):
    emb    = C[Xs]
    h      = torch.tanh(emb.view(-1, 6) @ W1 + b1)
    logits = h @ W2 + b2
    loss   = F.cross_entropy(logits, Ys)

    for p in parameters:
        p.grad = None
    loss.backward()
    for p in parameters:
        p.data += -0.1 * p.grad

Measured: 4.953627 → 0.007955 → 0.003903 → 0.002600 → 0.001965 — it reaches 0.008 in only 50 steps.

The value of the check is that it separates two classes of problem cleanly:

The logic is airtight: 5 samples, 3,481 parameters — the model is fully capable of memorising them. Failing to means the chain is broken somewhere.

That p.grad is None bug above would have surfaced in two minutes had this check been run first, instead of waiting for a full-batch run. From here on, this is the first thing to do when training anything.

Why 0.001965 and not 0

For the loss to reach 0, the probability of the right answer would have to be exactly 1 — which requires the corresponding logit to be +∞. Gradient descent can only approach it, decaying roughly as 1/step.

If those 5 samples happen to contain “the same context with different answers”, the loss has a floor above 0 — no model can predict two different answers at once.

Also, reset the parameters afterwards. The model just spent 200 steps memorising those five; it is overfitted to the extreme.

Mini-batch: three changes

for step in range(30000):
    ix = torch.randint(0, X.shape[0], (32,))     # ① draw 32 random indices

    emb    = C[X[ix]]                            # ② take only those 32 rows
    h      = torch.tanh(emb.view(-1, 6) @ W1 + b1)
    logits = h @ W2 + b2
    loss   = F.cross_entropy(logits, Y[ix])      # ③ the labels must be indexed too

③ is the one that gets forgotten. Indexing X[ix] but not Y gives a shape mismatch and an error — which is the lucky outcome; if the dimensions happen to line up it may not raise at all and simply be wrong.

Incidentally, the mine laid in the previous part goes off here for the first time: N drops from 228,146 to 32, and emb.view(228146, 6) is instantly out of range. Written as emb.view(emb.shape[0], -1) it is fine.

Measured side by side:

820× faster, and a considerably better result.

One detail worth noticing: the sample count fell by a factor of 7,130 but the speed rose only 820×. The difference is fixed overhead — the Python loop, randint, autograd bookkeeping — which no amount of shrinking the batch will save.

Why a “noisy” gradient does better

The gradient from 32 samples really is inaccurate; it is a noisy estimate of the full-batch gradient. But the point is this:

A gradient only has to be roughly right. Its job is to point a direction; you take one small step and compute it again. Veer off a little and the next step corrects for it.

exact direction, 1 step      vs      noisy direction, 7130 steps
    (full batch)                          (mini-batch)

Seven thousand noisy corrections travel far further than one perfect correction. The noise cancels out across steps.

Further: the noise is sometimes an asset, shaking parameters out of poor positions.

And there is no such thing as a full-batch gradient in modern deep learning, because the dataset does not fit in memory. GPT-3’s batch is 3.2 million tokens — still a minuscule pinch against a corpus of hundreds of billions.

The loss printed during training cannot be used

It comes from 32 samples and is extremely noisy:

step 400  loss 2.2671     ← these 32 happened to be easy
step 600  loss 2.5293     ← these 32 happened to be hard

Like estimating national average height from 32 people — the number jumps, and what jumps is the sample, not the people.

@torch.no_grad()
def evaluate(Xe, Ye):
    emb    = C[Xe]
    h      = torch.tanh(emb.view(-1, 6) @ W1 + b1)
    logits = h @ W2 + b2
    return F.cross_entropy(logits, Ye).item()

Evaluation = forward pass only, compute the loss, update nothing. Three differences from the training loop: no sampling, no backward, no update.

torch.no_grad() stops PyTorch from dutifully recording the computation graph (part 4’s _prev / _backward on the Value class). It still produces the right number without it, merely wasting a few hundred megabytes.

⚠️ But this is still evaluation on the training set. X/Y is the very data used to train, so it answers “how does the model do on data it has seen”, not “how does it do on names it has not”.

How much does that matter? Recall the 5-sample experiment — a loss of 0.001965 looks flawless, and that model is useless on any new sample. This number can only be read as “how much the model memorised”, not yet “how much it learned”.

How to read a loss curve

Plotting lossi directly gives a rectangle — the same width and height from end to end, no trend visible at all. Single-step loss bounces between 1.75 and 3.3 (a spread of 1.5) while the real improvement across 30,000 steps might be 0.1 — the noise buries the signal completely.

log10 does not help: it only compresses the vertical scale, and noise and signal are compressed equally. The signal-to-noise ratio is untouched.

Log plots earn their keep when something crosses orders of magnitude (a loss falling from 17 to 2.4). Here the loss stays within one.

A moving average does help:

li = torch.tensor(lossi)
n = len(li) // 200 * 200          # trim the tail so view() does not complain
smooth = li[:n].view(-1, 200).mean(1)

The noise drops by about 14× (√200 ≈ 14 — averaging n independent samples shrinks noise by √n), and the trend surfaces.

view(-1, 200) is the previous part’s view again, this time used to group: 30,000 numbers laid out as 150 rows × 200 columns, then averaged row-wise.

The second imprecision: sweeping the learning rate

The 0.1 in p.data += -0.1 * p.grad was picked out of the air, and there is evidence it is too large — three consecutive rounds of 30,000 steps:

The marginal return dropped forty-fold, and the smoothed curve is still dragging within ±0.02. This is not “finished learning” — it is steps too big to fit in, like coming down a mountain with strides so long that near the valley floor you keep bouncing off both slopes.

But “just make it smaller” is not the answer either; too small and training is unusably slow. There is a sweet spot between them, and it cannot be guessed.

Karpathy’s approach: make one run, but let the learning rate climb from tiny to huge across that run while recording the loss at every step.

lre = torch.linspace(-3, 0, 1000)      # the exponent: −3 → 0
lrs = 10 ** lre                         # the rate: 0.001 → 1.0

Why sample in the exponent

lre holds the exponent; lrs is the learning rate. What actually updates the parameters is lrs — lre is an intermediate value and the horizontal axis of the plot.

lre = learning rate exponent. That e is short for exponent, not Euler’s number — the base is whatever 10 ** in the code says it is.

Base 10 is chosen because people can do it in their heads: see −2 and you know it means 0.01. With e, −2 is 0.135 and you reach for a calculator. Switching to e gives identical results, only with the scale relabelled −6.9~0 and much harder to read. Rough division of labour: e for computing, 10 for showing.

The learning rate is multiplicative

First separate two “step sizes”: the sampling interval between adjacent linspace points (0.003 here, in the exponent), and the parameter update step gradient descent actually takes (lr × |grad|).

The learning rate is not the step size; it is a scaling coefficient — how far you actually move also depends on how large the gradient is. That also explains why the same rate behaves completely differently early in training (large gradients) and late (small ones).

The key is that the rate is multiplicative:

The same difference means entirely different things. So “how far apart” two learning rates are is a question of ratio, not of difference.

Each of the two samplings holds one quantity constant:

Which to choose depends on which quantity governs the effect. The rate is multiplicative → ratio governs → take the first.

Even spacing in the exponent is even ratios in the rate. The latter is what we want.

Where the points actually land:

Linear sampling wastes nine tenths of its points on 0.1—1.0, where adjacent points differ too little to matter, while the 0.001—0.01 range that may hold the answer gets 9.

The general rule: does it cross orders of magnitude

Every plot in the scaling-law papers is log-log for exactly this reason.

Reading the curve

The sweep code merely swaps lr for lrs[i] on each step, runs 1,000 of them, and takes 0.1 seconds.

The range [−3, 0] was chosen deliberately: below it nothing moves, above it everything diverges. The sign that the range was right is the left end already flat and the right end already turning up.

What a single point actually means

In the code, the loss is computed before the update with that rate. So the point at −0.5 on the horizontal and 3.0 on the vertical reads as:

By step 800 (having already walked the whole path of rates from 10⁻³ up to 10⁻⁰·⁵), the model’s current loss is 3.0.

It is not “the result of training with 0.32” — it is “what has accumulated by the time 0.32 is used”. And since the rate only increases, the horizontal axis is also a time axis.

What this method proves

This is the payoff of the part. The curve strictly proves exactly two things:

One: the loss turns upward at the right end. That is the only clean causal signal — having already fallen to 3, the loss should not climb back if the rate were harmless. It climbed, so those steps are destroying progress.

Two: the left stretch barely fell at all. The same number of steps later takes the loss from 12 to 3; this stretch moved it only from 20 to 12. So that band cannot move the model.

The long middle stretch proves nothing. The loss falls all the way to the valley — but that is fully explained by “training is progressing”. Fix the rate at 0.05 for all 1,000 steps and the loss would still go from 20 to about 3, on a curve of much the same shape.

“Wherever the minimum sits, that is the best rate” mistakes correlation for causation. Three defects: history dependence (every later point stands on the training done before it), each rate was tried for exactly one step, and the noise is large (every point is a single 32-sample batch).

So its place is that of a coarse filter:

before the sweep: lr ∈ [0.001, 1] (×1000)  →  after: [0.01, 0.4] (×40)

Its value is ruling out, not selecting. It excludes what is too slow and what will explode, and inside what remains no choice will be badly wrong. Its advantage is that it is cheap: faced with an unfamiliar model you may not even know the order of magnitude (0.001? 1? 100?), and 0.1 seconds answers that.

Why take the left of the valley

Not out of caution, but because the costs are asymmetric:

There is a second reason: the sweep measures starting from random initialisation, where gradients are large and a bigger rate is survivable. Later in training the parameters enter flat regions and the same rate looks too large.

Decaying the learning rate

Since it is too large late on, split the run in two:

Against the earlier table: a third round of 30,000 steps at 0.1 fell by 0.03. One third of the steps, more than double the fall.

The parameters had already reached the valley; steps of 0.1 vault over it onto the opposite slope and bounce. Shrinking by 10× is what lets them settle in.

⚠️ The ÷10 is convention, not something read off the curve. At 0.01 the curve still shows a loss around 10, which says training from scratch at 0.01 will not do; but the decay phase is a different situation (the parameters are already at the valley) and the curve cannot answer “by how much should it shrink”.

The main rate has a method (the sweep); the decay factor has a convention.

One sentence for this part: two deliberate imprecisions, with very different returns. Mini-batching trades a noisy gradient for 7,130× the steps, and that trade is clean; the sweep trades a coarse sampling for speed, but what it buys is only “ruling out”, never “selecting” — and using it to select is mistaking correlation for causation.

The loss now stands at 2.33, already past bigram’s 2.4544.

But every loss in this part was computed on the training set — it only answers how much the model memorised. The next part does something more basic first: turn one number into two. Without that ruler, all the tuning that follows is blind.