The Training Loop: Loss Wires All 41 Gradients Together

The last two parts built two components: an autograd engine, and a set of network classes. But so far MLP can only emit one random number — its parameters are randomly initialised and its output means nothing.

This part connects them: give it a target and let the network tune its own 41 parameters until it hits that target. Without this stretch, everything before it can compute but cannot learn.

There is still no new mathematics — squared error and gradient descent were both covered in part 2. The only new thing is joining them up.

The dataset

xs = [
    [2.0,  3.0, -1.0],
    [3.0, -1.0,  0.5],
    [0.5,  1.0,  1.0],
    [1.0,  1.0, -1.0],
]
ys = [1.0, -1.0, -1.0, 1.0]

Four samples, three dimensions each, targets of ±1.

Why ±1 rather than 0/1: tanh outputs in the range −1~1. The targets have to lie inside that range, or the network can never reach them.

The loss function

loss = sum((yout - ygt)**2 for ygt, yout in zip(ys, ypred))

This is why part 4 had to add __sub__ and __pow__ — subtraction and squaring earn their keep here for the first time.

Why square it: errors come with both signs, and adding them directly lets them cancel. Squaring makes every term non-negative, and punishes being badly wrong more than being slightly wrong.

loss is a Value: four outputs are gathered into one number, and the whole graph converges on a single endpoint.

Which resolves the loose end from the previous part — calling backward() on a single neuron’s output left the rest of the layer at zero. loss gathers every output into the same endpoint, and only then do all 41 parameters live in one graph.

The training loop

for k in range(30):
    # forward
    ypred = [n(x) for x in xs]
    loss = sum((yout - ygt)**2 for ygt, yout in zip(ys, ypred))

    # backward: zero first, then backward
    for p in n.parameters():
        p.grad = 0.0
    loss.backward()

    # update
    for p in n.parameters():
        p.data += -0.05 * p.grad

    print(k, loss.data)

The three parts are exactly the animation from part 2: forward for the cost → backward for the gradients → step along the negative gradient.

Note the minus sign in p.data += -0.05 * p.grad — the loss has to come down, so the step goes against the gradient.

Zeroing has to happen before backward(). The other way round (backward first, then zero) wipes the gradients that were just computed, and the parameters never move.

What happens if you deliberately skip the zeroing

Delete those two lines, change nothing else, and run both for the same 30 rounds:

The counterintuitive part: not zeroing descends faster

The two curves are different in kind:

• Zeroed slows as it goes, settling around 0.023. That is correct — the nearer the valley floor, the smaller the gradient, so the steps narrow on their own, exactly as in part 2’s convergence table
• Not zeroed accelerates, running all the way down to 9.26e−14, the floating-point floor

The reason is that gradients accumulate round on round, so the effective learning rate grows linearly:

the step at round k ≈ 0.05 × k × |∇|

By round 29 that is a learning rate of 1.45 — twenty-nine times what was set.

Running it again across ten random seeds

That curve is random.seed(3). Change the seed and the two variants stop being comparable at all:

The zeroed version lands in the same narrow band all ten times; the non-zeroed one spans twenty-six orders of magnitude, and three of those runs plunged to around 1e−16 before being thrown back out (one rebounded to 0.69 — thirty times worse than the zeroed version).

The reason this example so often “gets away with it” is that three conditions all happen to hold: four simple samples whose gradients do not conflict, so accumulating is just acceleration in a consistent direction; 41 parameters for 4 samples, so badly over-parameterised that almost any path reaches the target; and targets of ±1 sitting exactly on tanh’s asymptotes, so the task inherently rewards large steps. Raise the learning rate while still not zeroing and the oscillation shows up immediately.

The real damage is not that it fails

It is that you lose control:

That seed table above is this column measured. To descend faster, tune the learning rate rather than relying on a bug.

A loss of 0 does not mean it learned

“Isn’t a smaller and smaller loss exactly what we want?”

Training loss is not the goal; it is a proxy. The goal is performing well on data it has never seen.

This example has 4 samples and 41 parameters — ten times more parameters than samples. It is entirely capable of memorising four answers rather than learning any rule.

The extreme comparison: store the training set in a dictionary and look it up, and the training loss is exactly 0 — while being useless on any new input. A loss of 0 does not mean it learned; it may only mean it remembered.

The classic sign of overfitting is the two curves parting company:

training loss:    0.5 → 0.1 → 0.01 → 0.001   ↓ keeps falling
validation loss:  0.6 → 0.3 → 0.35 → 0.52    ↑ starts climbing

The moment validation loss bottoms out and turns back up is the moment to stop.

An instinct worth building: a loss falling unusually fast is more often a bug than a triumph. Not zeroing, data leakage, the evaluation set bleeding into training — they all present as results that are too good to be true. That curve running down to 1e−14 is a live specimen.

A Jupyter trap

A freshly built n = MLP(...), and yet the loss came out at 0.07 — the level of a trained network.

The cause: ypred was still the old variable from the end of the previous training run, never recomputed after the new network was built. Variables survive across cells in memory; they do not refresh because some other cell was re-run. Change something upstream and everything downstream has to be re-run.

A good habit: print the predictions once before training and once after. The comparison is what makes it convincing.

In short

The training loop connects the parts from the last two: loss gathers every output into one number, giving the graph a single endpoint; backward() fills in all 41 gradients; the parameters step along the negative gradient. Repeat those three stages a few dozen times and the network has learned.

And the debt that += ran up is that every round must start by zeroing.

That completes V1, with three things built from scratch and depending on no framework at all: a complete autograd engine (Value), a set of network classes (Neuron / Layer / MLP), and a training loop that works.

Next is V2: makemore bigram. Real data at last (a names dataset), a character-level language model, softmax, cross-entropy loss.

That exp operator added back in part 4 was groundwork for softmax. From that episode on, things shift gradually towards tensor operations.