Deep nets change the distribution of each layer's inputs as weights update, which the authors called internal covariate shift. Their fix: for each activation channel, subtract the minibatch mean and divide by the minibatch standard deviation, then learn a scale and shift (γ, β) so the layer can still represent the identity. Ioffe and Szegedy showed faster training and higher learning rates on ImageNet-class CNNs. Whether the theoretical story about covariate shift is the full explanation has been debated; the engineering win is not in doubt.
BN also regularizes a bit because each example's normalization depends on its batch-mates. That is delightful until your batch size is 2.
Train stats versus population stats
During training you use batch moments. For inference you use running averages collected during training (or a dedicated pass). If you leave the layer in train mode at eval, accuracy jitters with batch composition. If you never update running stats (a frozen BN in a tiny fine-tune), you will normalize with the wrong mean. Detection and segmentation codebases are graveyards of "BN in eval vs train" bugs.
Small batches make the batch mean a bad estimator. That is why GroupNorm, LayerNorm, and InstanceNorm exist. Transformers standardized on LayerNorm (and later RMSNorm) because the batch axis is not a reliable statistic when you pipeline or use batch 1 decode. BN is a CNN-era default, not a universal law.
A worked fine-tune trap
You fine-tune a BN-filled ResNet with batch size 4 per GPU, no sync-BN. Each GPU sees a biased mean. Val in train mode looks okay; production batch-1 inference uses running stats from those noisy updates and drops. Fix: freeze BN stats (eval for BN modules) or use SyncBatchNorm or replace with GN. The paper assumed healthy batch sizes.
Failure modes
BN + dropout stacked without a sweep.
Virtual-batch or ghost-BN hacks that you do not replicate at export.
Applying BN to sequences across time naively so future frames leak (video).
Forgetting γ and β and forcing unit Gaussian activations forever.
SyncBN, GhostBN, and export
Multi-GPU training with per-replica batch 2 is not the paper's BN. SyncBatchNorm averages across devices so the statistic matches a larger logical batch; it costs a communication. Frozen BN (eval stats, trainable last layers) is the detection-fine-tune default. When you export to ONNX, fold γ,β and running stats into a scale-bias when possible so mobile runtimes do not implement BN wrong. If the folded graph and the train graph disagree on a unit test image, stop shipping. BN bugs are silent pixel shifts, not crashes.
Do not mix FrozenBN and trainable BN in one model without writing it down. A neck with frozen stats plus a new head with live stats is a valid detection trick; an accidental mix from a copy-paste module is not. Unit-test training flags on export.
What you can borrow
- Normalize per channel with learned affine parameters to stabilize deep CNN training.
- Ship population statistics; never serve BN in train mode unless you mean MC noise.
- Treat batch size as part of the BN spec; switch to LN/GN when the batch is tiny or sequential.
- Freeze BN stats when fine-tuning with small batches.
- Do not paste BN into transformers as a default; LayerNorm is the culture there for a reason.