ELECTRA: Pretrain by Detecting Replaced Tokens, Not Only by Masking Them
Clark et al. train a discriminator to tell original tokens from plausible replacements a small generator inserted. Sample-efficient encoder pretraining, and when MLM is still simpler.
3 min read