Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization

Yao Xiao, Hai Ye, Linyao Chen +4

Iterative data generation and model retraining are widely used to align large language models (LLMs). It typically involves a policy model to generate on-policy responses and a rew…