Iterative Label Refinement Matters More than Preference Optimization under Weak Supervision

Yaowen Ye, Cassidy Laidlaw, Jacob Steinhardt

Language model (LM) post-training relies on two stages of human supervision: task demonstrations for supervised finetuning (SFT), followed by preference comparisons for reinforceme…