Gradients and Backpropagation: Three Rounds with One Cat

Separate the three actions first

“Backpropagation” and “gradient descent” are often treated as one thing. They are two:

• Backpropagation is the algorithm that computes gradients. It answers “which direction should each parameter move, and by how much”, and then it is done.
• Gradient descent is the policy that uses those gradients. Given them, it nudges the parameters one step along the negative gradient.

One round of training is three actions in sequence: forward to get the cost → backward to get the gradients → update the parameters. Backpropagation is only the middle one.

The animation below splits those three actions into 22 steps you can walk through one at a time. Every number in the diagram can be clicked to see how it was worked out.

The network is tiny: two inputs (ears 0.9, eyes 0.8), two hidden neurons, one output — nine parameters in all. The target is y = 1.0 (this is a cat).

Forward: z is only a way-station

The two steps inside every neuron (covered in part 1; repeated here for reference):

z = w₁·x₁ + w₂·x₂ + b
a = σ(z) = 1/(1+e⁻ᶻ)

In round one, h1 computes 0.5×0.9 + 0.5×0.8 + 0 = 0.85, then σ(0.85) = 0.7006.

What goes on to the next layer is a, not z. When the output layer computes z2 it multiplies by 0.7006 (h1’s a), not by 0.85. z is only an intermediate; the reason it exists at all is that backpropagation will need its slope later.

Three layers in, the network outputs a2 = 0.6129 — “61.3% confident it is a cat”.

The cost only tells you how wrong you are

C = (a2 − y)² = (0.6129 − 1.0)² = 0.1498

Forward propagation ends here. But C is just a number: it says how far off the network is, not what to change. What to change is the next step’s job.

Where backward starts: δ

Turn around. The first thing to compute is the output layer’s δ, the product of two terms:

∂C/∂a2 = 2(a2 − y) = 2 × (0.6129 − 1) = −0.7741     how far from the target
σ′(z2) = a2(1 − a2) = 0.6129 × 0.3871 = 0.2372      can the signal get through the activation
δ_out  = −0.7741 × 0.2372 = −0.1837

δ is defined as ∂C/∂z — how sensitive the cost is to this layer’s weighted sum. It is worth computing on its own because all three parameters in this layer reuse it, rather than each deriving it separately.

The baton: what travels back is a demand, not a gradient

On the way back there are two kinds of multiplication, and the rule is simple: crossing an edge multiplies by that edge’s weight; crossing a neuron multiplies by its slope.

h1's baton: δ_out × w2_1 = −0.1837 × 0.7   = −0.1286
h1's δ    : −0.1286 × σ′(z1_1)=0.2098      = −0.0270

That intermediate −0.1286 is ∂C/∂a1_1. It is not any parameter’s gradient. It is the output layer’s demand on the previous layer — the output layer asking that the previous layer’s activation move in a certain direction, from which that layer works out how its own parameters should change.

It stops at the input layer: pixel values are given, so they need no gradient.

Whether a weight is worth adjusting comes down to three factors

The formula from Ch.4, unpacked, is really one sentence:

∂C/∂w = a_prev · σ′(z) · 2(a − y)
          ↑         ↑        ↑
      how strong  can it   how far
      the signal  get past  from the
      from below  the act.  target

If any one of the three is near 0, that weight’s gradient is near 0 and it barely needs to move.

• the neuron below did not fire (a_prev = 0) → whatever you do to this weight gets multiplied by 0
• the neuron is saturated (σ′ ≈ 0) → the signal cannot get through; the chain breaks here
• it is already right (a = y) → there is nothing to fix

The second one is the source of vanishing gradients. Sigmoid’s slope peaks at just 0.25 (at z=0) and flattens to nearly 0 at both ends.

Two things you only see once you run it

Everything above follows from the formulas. The next two only show up when you run three full rounds and watch the real numbers.

One: the error signal flips sign through a negative weight

δ_out = −0.1837 is negative. Passed back to h1 it gives −0.0270, still negative; passed to h2 it becomes +0.0179.

The reason is that w2_2 = −0.4 is a negative weight:

h2's baton: −0.1837 × (−0.4) = +0.0735   ← two negatives make a positive
h2's δ    : +0.0735 × 0.2441  = +0.0179

So one and the same demand — “the output is too low, push it up” — arrives at the two hidden neurons as opposite instructions: h1 should go up, h2 should go down. You cannot see this in a single-layer network, because the error signal only travels one hop.

Two: the earlier the layer, the smaller the gradient

Round one’s nine gradients, grouped by layer:

The hidden layer is roughly 7× smaller than the output layer. The reason is the baton step:

δ_h1 = δ_out  × w2_1 × σ′(z1_1)
     = −0.1837 × 0.7  × 0.2098   = −0.0270
       ↑         ↑      ↑
    signal    crossing  crossing
    from      an edge:  a neuron:
    above     × weight  × slope

Every layer you travel back multiplies in two numbers below 1 — one weight, one slope. Here that is 0.7 × 0.2098 = 0.147, so each layer shrinks the signal to about 1/7.

The σ′ factor is unavoidable. Sigmoid’s slope peaks at 0.25 (at z = 0) and approaches 0 towards both ends:

So travelling back through a neuron multiplies in a factor of at most 0.25. That is fixed by the shape of the sigmoid.

What “learns slowly” actually means

Back to the update rule: new = old − η × gradient. The gradient is the step size.

• output layer gradient 0.18 → the parameter moves 0.18
• hidden layer gradient 0.027 → it moves only 0.027

In the same round of training, the later layers have already shifted substantially while the earlier ones have barely moved. It is not that the earlier layers do not need to learn — it is that they never receive a signal strong enough to learn from.

With enough layers it stops being “slow” and becomes “stopped”

Compounding that 1/7 per layer on the way back:

By a dozen or so layers, the frontmost layer’s gradient is down to the order of 10⁻⁹. Multiplied by the learning rate and added to a parameter, in float32 that is essentially adding nothing — the first few layers are frozen, and no amount of further training moves them.

This is the vanishing gradient: nothing blocks the signal, it is simply multiplied away layer by layer on the trip back.

This is also the main reason ReLU is now used far more than sigmoid. ReLU’s derivative on the positive side is 1, not ≤ 0.25 — multiplying by 1 costs nothing, so the signal can travel all the way back to the front.

Update: this is the step that actually changes the network

Backpropagation hands over nine gradients and stops; the network has not changed at all yet. This is the step that acts:

new = old − η × gradient        (here η = 1)
w2_1 : 0.700 − (−0.1287) = 0.829

All the gradients are negative, which means “increasing these lowers the cost”, so every parameter grows after the update.

After three rounds

The drop shrinks each round. Not because the learning rate changed (η stays 1), but because the closer to the valley floor, the smaller the gradient itself, so the step η·∇C narrows on its own.

Which is why training does not end at “cost equals 0” — sigmoid only outputs 1 as its input goes to infinity, so a perfect prediction is mathematically out of reach. The real end is “the gradient is about 0, so there is no point going further”. In practice that means early stopping: once the cost has all but stopped falling, stop.

That is training

The three rounds do exactly the same thing; only the parameters differ. A real network just repeats this loop tens of thousands of times with hundreds of millions of parameters instead of nine — every step does precisely what happens here.

One thing about the network above is special, though: its output layer has only one neuron. So each hidden neuron has exactly one downstream, and nothing has to be merged on the way back.

Once the output layer has two neurons, one extra step appears. That is the next part.