A Thousandfold Range In, 1.000 Out: What BatchNorm Pays for It

Part 14’s lede and its final section both said “BatchNorm is the next section”, and part 15’s closing already gave away the mechanism — “subtract the mean and divide by the standard deviation at every layer’s output, regardless of what happened upstream”. This part delivers on it.

In one line: initialisation only governs step 0; BatchNorm governs every step.

And it is the one operation in the whole network where samples affect each other — which is both its power and the source of all its trouble. Those two things take up roughly half this part each.

Kaiming predicts, BatchNorm measures

Part 14’s last section already explained why Kaiming stops holding: it assumes the input statistics are known and stable, but layer 2’s input is layer 1’s output, so as soon as upstream moves, everything computed downstream is void. Part 15 traced it to the root — the formula rests on “W independent of the data” and “zero mean”, and both start decaying the moment training begins.

This part won’t repeat either. Just one contrast in framing:

How it satisfies two goals that were in conflict

Part 14 left a promise unpaid: “avoiding saturation wants z small, avoiding decay wants z large. Kaiming can only compromise — what actually solves both at once is BatchNorm.”

Kaiming has exactly one knob, σ_W, and that knob controls both things at once, so a compromise is all it can offer. BatchNorm splits it into two independent knobs:

1. Every step, z is pulled to σ = 1 on the spot — non-saturation becomes unconditional, whatever σ_W happens to be
2. Then γ puts back the width for whichever neurons genuinely need it — how much variance is needed is something the network learns

The compromise disappears because “guaranteeing the scale” and “deciding the scale” are no longer the same parameter’s job.

Two steps, one job each

A word first. In this series “normalisation” has meant softmax’s divide by the total (making a set of numbers sum to 1). BatchNorm does something else — subtract the mean, divide by the standard deviation, so a set of numbers has mean 0 and standard deviation 1. Below this is called standardising, and BatchNorm is abbreviated BN.

ẑ = (z − μ_batch) / σ_batch

μ and σ are 200 numbers, not one. The next section is entirely about that.

“Standardising” looks like one step. It is two, and only the second one fixes anything.

⚠️ The note this is based on says saturation is “still 47%” after subtracting the mean. That does not hold. Sliding pushes points across the saturation line, and centring is in fact the minimum of the saturated fraction for a fixed σ (48.2% at μ=1.8, 41.4% once centred).

But its conclusion is right, and more valuable: subtracting the mean cannot change σ, and σ is what sets the saturated fraction — what takes 41.4% down to 0.8% is dividing by the standard deviation.

Subtracting the mean has its own separate justification: tanh is symmetric about the origin, and sitting to one side uses only half of it.

Both steps are affine, so neither changes the shape of the distribution. A bimodal input comes out bimodal; only the centre moves to 0 and the width becomes 1. (z is already close to normal — that is part 15’s central limit theorem, and has nothing to do with BN.)

Why the target is μ=0, σ=1

The second is the important one: part 14’s table of “ratio 1.5 per layer → 57× after ten” is exactly this.

Which axis

z.mean(0)     # along the sample axis → (200,)  ✅
z.mean(1)     # along the neuron axis → (32,)   ❌
z.mean()      # everything as one pile → scalar ❌

Statistics go per neuron, across every sample in the batch. The 200 neurons have learned entirely different features; averaging them together means nothing.

This is also the difference between W and z

Part 15’s closing line that “the standard deviation of a matrix is a lazy phrase” holds only for W — because randn generates each element independently, so it slices the same way in every direction. z does not; its two axes are two different things.

Measured:

before BN, std per column: [3.712, 6.196, 3.600, 6.854, 5.480]   ← all different
after  BN, std per column: [1.0,   1.0,   1.0,   1.0,   1.0  ]   ← levelled

γ and β: a learnable undo switch

Standardising alone would take away the network’s right to decide its own output distribution, so a learnable restoration follows:

y = γ · ẑ + β

Initially y = 1·ẑ + 0 = ẑ, which is doing nothing at all. But they are parameters, and training pushes them.

They are ordinary parameters, with no magic

The derivatives (summed over the 32 samples, since one batch shares one set of γβ):

∂L/∂γⱼ = Σᵢ (∂L/∂yᵢⱼ) · ẑᵢⱼ          ∂L/∂βⱼ = Σᵢ (∂L/∂yᵢⱼ)

γ corresponds to a weight, β to a bias, and the rules are copied straight from a linear layer — they are part 13’s two expressions.

Measured: they really do get trained away

bngain: mean 1.216   range [0.02, 3.08]
bnbias: mean −0.016  range [−0.87, +1.59]

They start as all 1s and all 0s, and by the end γ spans a factor of 154. That answers directly whether forced standardising breaks training:

γ close to 0 amounts to automatic pruning — some model-compression methods work precisely by finding channels with γ ≈ 0 and deleting those neurons.

Why it isn’t wasted work

Standardising is not a lock; it is a change of parameterisation:

Downstream layers always see a standard input distribution, and are never forced to relearn just because something upstream was nudged.

⚠️ The mechanism is still disputed. Part 14 named internal covariate shift; that was the 2015 original paper’s explanation. A 2018 MIT paper refuted it, arguing that what actually helps is BatchNorm smoothing the loss surface, with little to do with distributional drift.

Its effectiveness is agreed on; its mechanism is not settled. This section is about what it does, not why it works — the latter has no answer yet.

b1 disappears

((z + b) − mean(z + b)) / std(z + b) = (z − mean(z)) / std(z)

Whatever constant b1 adds is subtracted right back out by the mean, so it has no effect on the result and its gradient is 0. Shifting is bnbias’s job now.

A layer with BN needs no bias. In ResNet’s Conv → BN → ReLU arrangement the convolutions all carry bias=False — for this reason, not as some optimisation trick.

Sample coupling

Only BN breaks sample independence in the whole network:

Measured:

# inference mode: the same sample twice
a = forward(Xtr[:1], training=False)
b = forward(Xtr[:1], training=False)
torch.allclose(a, b)        # True  ✅

# training mode: the same sample in different batches
r1 = forward(cat([Xtr[:1], Xtr[100:131]]), training=True)[0,0]   # −3.8143
r2 = forward(cat([Xtr[:1], Xtr[500:531]]), training=True)[0,0]   # −4.3548

Same sample, same parameters, different company — the output differs by 0.54.

⚠️ In a medical setting that is unacceptable: the same X-ray, sent in alongside a different set of X-rays, produces a different diagnosis.

But the bug turns out to be a feature

The random noise that coupling introduces acts as regularisation — which is why models using BN often need less dropout, or none. A side effect introduced to stabilise training quietly solves a different problem.

What about inference

Part 11 gave a definition of evaluation: evaluation = forward only, compute the loss once, update nothing — three differences from the training loop: no sampling, no backward, no update. That definition needs one more line here — BatchNorm is the first thing in this series that makes the forward pass itself differ between training and inference.

With one sample, BN simply breaks

At inference there is often exactly one sample (generate one name, a user uploads one scan):

z1        = [−1.406, −2.973, 0.255, 2.261]
z.mean(0) = [−1.406, −2.973, 0.255, 2.261]   ← the mean is the value itself
z.std(0)  = [nan, nan, nan, nan]              ← one number has no standard deviation

BN is the only place in the whole chain that jams. x @ W1, tanh and softmax all handle a single sample perfectly well — because they are per-sample independent. This is a direct consequence of the previous section.

The fix: keep a running statistic while training

bnmean_running = torch.zeros((1, n_hidden))   # not a parameter, a record
bnstd_running  = torch.ones((1, n_hidden))

with torch.no_grad():                          # ❗ stays out of the graph
    bnmean_running = 0.999 * bnmean_running + 0.001 * zmean
    bnstd_running  = 0.999 * bnstd_running  + 0.001 * zstd

Each step keeps 99.9% of the history and mixes in 0.1% of the current value. Expanded, it is a weighted average:

The effective window is 1/(1−0.999) = 1000 steps = 32,000 samples, far steadier than a single batch of 32.

A plain average is not used because it would remember the values from early training — at step 1 the model had just been initialised and that distribution is long obsolete. An EMA forgets old data automatically and drifts along with training.

Same trick as part 11’s smoothing of the loss curve: a single observation is noisy, so take a moving average to get a stable estimate.

At inference, fixed values

def forward(X, training=True):
    global bnmean_running, bnstd_running
    z = C[X].view(X.shape[0], -1) @ W1

    if training:
        zmean = z.mean(0, keepdim=True)
        zstd  = z.std(0, keepdim=True)
        with torch.no_grad():
            bnmean_running = 0.999 * bnmean_running + 0.001 * zmean
            bnstd_running  = 0.999 * bnstd_running  + 0.001 * zstd
    else:
        zmean, zstd = bnmean_running, bnstd_running    # ← fixed

    z = bngain * (z - zmean) / (zstd + 1e-5) + bnbias
    return torch.tanh(z) @ W2 + b2

At inference BN degenerates into a fixed affine transform, and samples become independent again — that 0.54 difference from the previous section disappears.

This is what model.train() / model.eval() are

❗ Forgetting to call model.eval() is one of the most common bugs in deep learning. Symptoms: a suspiciously low validation loss, the same sample predicting differently twice, results changing when the batch size changes. Dropout responds to the same two switches.

Four sets of numbers survive training, per BN layer

But both must be saved — state_dict() contains both kinds. Save γβ without the running values and inference is completely broken after loading.

BN does not win on this network

The standardising itself works: z.std goes from 1.597 to 0.984 and saturation drops to 0.8%. But the four versions differ by at most 0.0146, entirely inside the noise.

The parameter accounting is worth doing properly. Two numbers here look like they disagree:

γ and β, 200 each         → +400
b1 becomes useless here   → −200
11897 − 200 + 400 = 12097   → net +200

“The cost is 400 parameters” and “the table only shows +200” are both right, about different things — the “b1 disappears” section above is what settles the account.

Why it doesn’t win

The problems it exists to solve — signal decay in deep networks, initialisation being hard to tune, distributions drifting between layers — none of them exist in a 2-layer network, while the cost (an extra statistic every step, batch noise, a second set of numbers to keep for inference) is paid immediately.

That is the third time this episode that a technique designed for depth pays nothing on a shallow network. The first two were Kaiming losing to a hand-picked 0.2, and widening the hidden layer making things worse.

Its real value is elsewhere

Three initialisations spanning a factor of a thousand, measured:

A thousandfold difference in, identical out. That is the whole meaning of “rather than getting it right, force it” — part 14 spent an entire article tuning σ_W, and BN turns that into something that does not need tuning.

A corollary: after BN, the length of W1 carries no meaning, only its direction does — the scale gets eaten by the standardising. It is also why weight decay works differently on layers that have BN.

Two side effects

① It is a form of regularisation — the noise from sample coupling behaves like dropout, as above.

② It fails when the batch is too small — at batch = 2 the statistics are extremely noisy and BN does more harm than good.

The second is the direct reason LayerNorm exists: it standardises along the feature axis (the horizontal direction marked ❌ in the figure above), which is independent of batch size and introduces no sample coupling.

The price is giving up cross-sample statistics as a source of information — but for sequence models each position’s features should be handled independently anyway, so that price is not really paid. Transformers use LayerNorm throughout, never BatchNorm.

The accounting here is clear: BN takes away sample independence and returns initialisation no longer mattering. On a 2-layer network that trade is a bad one — the problems it solves have not shown up yet, and the cost is paid in full. But every item in that cost — running statistics, model.eval(), a batch that cannot be too small — is not a patch. Each one follows necessarily from the single decision to let samples affect each other.

In part 14’s checklist, step ② was checking the initial loss and step ③ the per-layer σ ratio. After BN both of those answers are guaranteed unconditionally — the checklist has not become wrong, its first two rungs have simply been made redundant.