InstructGPT: Alignment as a Training Pipeline, Not a Prompt Preface
Ouyang et al. turned human preference data into a reward model and PPO updates so a smaller GPT-3 followed instructions better than a larger base model. The pipeline, the costs, and the failure modes still show up in every RLHF stack.
3 min read