Deep nets change the distribution of each layer's inputs as weights update, which the authors called internal covariate shift. Their fix: for each activation channel, subtract the minibatch mean and divide by the minibatch standard deviation, then learn a scale and shift (γ, β) so the layer can still represent the identity. Ioffe and Szegedy showed faster training and higher learning rates on ImageNet-class CNNs. Whether the theoretical story about covariate shift is the full explanation has been debated; the engineering win is not in doubt.

BN also regularizes a bit because each example's normalization depends on its batch-mates. That is delightful until your batch size is 2.

Train stats versus population stats

During training you use batch moments. For inference you use running averages collected during training (or a dedicated pass). If you leave the layer in train mode at eval, accuracy jitters with batch composition. If you never update running stats (a frozen BN in a tiny fine-tune), you will normalize with the wrong mean. Detection and segmentation codebases are graveyards of "BN in eval vs train" bugs.

Small batches make the batch mean a bad estimator. That is why GroupNorm, LayerNorm, and InstanceNorm exist. Transformers standardized on LayerNorm (and later RMSNorm) because the batch axis is not a reliable statistic when you pipeline or use batch 1 decode. BN is a CNN-era default, not a universal law.

A worked fine-tune trap

You fine-tune a BN-filled ResNet with batch size 4 per GPU, no sync-BN. Each GPU sees a biased mean. Val in train mode looks okay; production batch-1 inference uses running stats from those noisy updates and drops. Fix: freeze BN stats (eval for BN modules) or use SyncBatchNorm or replace with GN. The paper assumed healthy batch sizes.

Failure modes

BN + dropout stacked without a sweep.

Virtual-batch or ghost-BN hacks that you do not replicate at export.

Applying BN to sequences across time naively so future frames leak (video).

Forgetting γ and β and forcing unit Gaussian activations forever.

SyncBN, GhostBN, and export

Multi-GPU training with per-replica batch 2 is not the paper's BN. SyncBatchNorm averages across devices so the statistic matches a larger logical batch; it costs a communication. Frozen BN (eval stats, trainable last layers) is the detection-fine-tune default. When you export to ONNX, fold γ,β and running stats into a scale-bias when possible so mobile runtimes do not implement BN wrong. If the folded graph and the train graph disagree on a unit test image, stop shipping. BN bugs are silent pixel shifts, not crashes.

Do not mix FrozenBN and trainable BN in one model without writing it down. A neck with frozen stats plus a new head with live stats is a valid detection trick; an accidental mix from a copy-paste module is not. Unit-test training flags on export.

What you can borrow

  • Normalize per channel with learned affine parameters to stabilize deep CNN training.
  • Ship population statistics; never serve BN in train mode unless you mean MC noise.
  • Treat batch size as part of the BN spec; switch to LN/GN when the batch is tiny or sequential.
  • Freeze BN stats when fine-tuning with small batches.
  • Do not paste BN into transformers as a default; LayerNorm is the culture there for a reason.