BERT masks tokens; images have more redundancy, so a 15% mask is too easy — the net inpaints from nearby texture. MAE masks a high fraction of patches (they like 75%), runs a ViT encoder on the visible patches only, then a small decoder that sees encoded patches plus mask tokens and predicts pixel values. He and colleagues get a sample-efficient, compute-efficient pretrain: the encoder never looks at mask tokens until after pretraining, so it processes a quarter of the sequence.

After pretrain you throw the decoder away and fine-tune or linear-probe the encoder. That is the product: an initialization, not an inpainting demo, even though the reconstructions are the debugging vis.

Asymmetry is the systems trick

A heavy encoder on few tokens and a light decoder on the full set (with masks) is how they keep FLOPs down. If you encode all patches plus masks like a naïve BERT-vision, you lose the win. If you mask only 15%, the task is too local. The high mask ratio is conceptual, not a cute number: it forces global reasoning.

Pixel reconstruction is a simple loss (they use MSE on pixels, with optional per-patch normalization). Contrastive methods need big batches and careful augmentations. MAE needs a decoder and a mask sampler. Different production constraints. If you cannot hold SimCLR batches, MAE is friendlier. If you need invariance to a specific aug (color jitter for your domain), contrastive might still match the product better.

Fine-tune protocol matters

They report linear probes and end-to-end fine-tunes. A weak probe does not mean a weak encoder. Use the paper's fine-tune recipe (LR, layer decay, epochs) before you abandon MAE. Detection and segmentation need adapters (FPN on ViT) — MAE does not emit a pyramid by itself.

A worked mask bug

You sample random patches but accidentally leak mask positions into the encoder via a bad implementation that inserts mask tokens early. Compute goes up, accuracy behaves like a different paper. Unit-test that encoder sequence length is (1 - mask_ratio) * N. Visualize reconstructions: if they are too sharp too early, the task may be too easy (mask ratio too low).

Failure modes

Decoder left attached at fine-tune, wasting RAM, or encoder not receiving the right weights.

MSE on a dataset where semantics ≠ pixels (you reconstruct lighting, not objects). Try a HOG or perceptual target only if you measure.

Tiny data pretrain that memorizes images; MAE is not magic few-shot.

Comparing to supervised ViT pretrained on JFT as if the data matched.

Transfer beyond classification

MAE encoders feed detectors and segmenters via ViT adapters (simple FPN, ViTDet). Pixel pretraining does not magically create multi-scale maps; you still build a neck. If detection mAP is weak after a great ImageNet fine-tune, the neck and resolution, not the MAE loss, are the first suspects. Keep a reconstruction visualization in the training job: if it stops looking like the image family you care about (x-rays vs ImageNet), your pretrain domain is wrong.

What you can borrow

  • Mask aggressively so reconstruction cannot be local copy-paste.
  • Encode visible patches only; keep the decoder small and disposable.
  • Use MAE when you want ViT inits without contrastive batch taxes.
  • Follow a known fine-tune recipe before judging the encoder.
  • Prefer contrastive or supervised pretrain when your invariances are specific and labels or augs already encode them.