Anatomy of Post-Training: Using Interpretability to Characterize Data and Shape the Learning Signal

Leon Bergen, Usha Bhalla, Sidharth Baskaran +14

Language-model post-training is the main stage at which model behavior is shaped, yet it still largely involves optimization of scalar rewards that summarize diverse desiderata. Th…