Multi-Path Summation: The Σ a Single-Output Network Hides

Only one thing changes in the structure

The hidden layer is untouched; only the output layer goes from 1 neuron to 2 — one judging cat, one judging dog. Parameters go from 9 to 12.

The target goes from one number to a pair: y = (cat 1.0, dog 0.0). The cost is correspondingly a sum of two terms:

C = (a2_1 − y₁)² + (a2_2 − y₂)²
  = (0.6129 − 1.0)² + (0.4740 − 0.0)²
  = 0.1498 + 0.2247 = 0.3745

Note that first term, 0.1498 — it is exactly the entire round-one cost of the single-output network in part 2, because the parameters along the cat branch are identical to the ones there.

The consequence: h1 now has two downstreams

That is the only change, but it changes backpropagation.

In part 2, h1 connected to one output, and the error signal came back from that one place. Now h1 feeds both the cat and the dog neuron, and both make demands on it — demands that may disagree.

So which one should h1 listen to?

The plus sign comes with the rule; it was not bolted on

This plus sign is not a convention along the lines of “several paths, so add them up”. It is what the multivariable chain rule gives directly:

If C depends on x through intermediates u₁, u₂, …, then ∂C/∂x = Σⱼ (∂C/∂uⱼ)·(∂uⱼ/∂x)

Here, two intermediates use a1_1 directly: z2_1 (the cat’s weighted sum) and z2_2 (the dog’s). Substituting:

∂C/∂a1_1 = (∂C/∂z2_1)·(∂z2_1/∂a1_1) + (∂C/∂z2_2)·(∂z2_2/∂a1_1)
         = δ_o1 × w2_11              + δ_o2 × w2_21

As many direct downstreams, as many terms. With 10 neurons in the output layer this would be a sum of 10.

The number from part 2 is the first term of this sum

This is what aligning the two animations buys — you can read off exactly what got added.

Round one, h1’s two paths:

via cat: δ_o1 × w2_11 = −0.1837 × 0.7    = −0.1286   ← part 2's baton, identical
via dog: δ_o2 × w2_21 = +0.2364 × (−0.5) = −0.1182   ← the new one
                                     total = −0.2467

In part 2, h1’s baton was −0.1286; here it is −0.2467, roughly double. Everything extra comes entirely from the new dog path.

So a single-output network does not lack the summation — its summation just has one term. It is the n=1 special case, where you cannot tell there is a sum at all.

Signs: this time the two paths point the same way

The signs are worth a look.

• δ_o1 = −0.1837 (negative): the cat output, 0.613, is below its target of 1.0, so it needs to go up
• δ_o2 = +0.2364 (positive): the dog output, 0.474, is above its target of 0.0, so it needs to come down

The two δ have opposite signs, yet after reaching h1 both terms are negative and stack into a larger negative number. The weights are why:

cat: negative δ × positive weight (+0.7)  = negative
dog: positive δ × negative weight (−0.5)  = negative     ← the weight flipped the sign

“The error signal flips sign through a negative weight”, from part 2, is here the reason the two paths can reinforce each other. They are the same fact stated twice: the sign is set by the weight, not by δ.

On h2’s side it is the mirror image — both terms are positive (+0.0735 and +0.1418), totalling +0.2153.

After the sum, everything is as before

Once the batons are merged, the remaining steps are word for word what they were with one output:

δ_h1 = −0.2467 × σ′(z1_1)=0.2098 = −0.0518
then multiply by x1 / x2 / 1 for this layer's three gradients, and stop at the input layer

After three rounds:

Both outputs move toward their own targets without fighting each other.

In short

The only difference from a single-output network is turning the baton from one term into a sum of several. Every other step — computing δ, multiplying by the slope, by the activation, updating the parameters — is identical.

Real networks have hundreds of neurons per layer, so the sum goes from 2 terms to several hundred, and nothing about its nature changes. Matrix multiplication is exactly this operation: Σⱼ δⱼ·wⱼ written in matrix form is the transpose of the weight matrix times the δ vector.

At this point the first four 3B1B chapters have all been grounded in numbers you can run. Next is Karpathy’s micrograd: implementing a Value class and automatic differentiation in code, where all these hand-computed gradients come straight out of .backward().