Initialisation only governs step 0; BatchNorm governs every step — vary σ_W by a factor of a thousand and the output after BN is identical. The price is that it becomes the one operation in the network where samples affect each other, which is why inference needs its own statistics, and why model.eval() exists.
The previous part borrowed σ_z = √n · σ_W · σ_x and moved on. This one derives it from the definition of expectation: three rules of arithmetic, why the cross term vanishes, why it is variance and not standard deviation that adds — and why diagnosing a network means watching the variance rather than the mean or the maximum.
An initial loss of 17 and six in ten neurons dead on arrival — because the standard deviation is amplified once at each of two matrix products. The fix is four lines, but what actually travels is the checklist of what to check in what order, and why Kaiming loses to a guessed number on this particular network.