Staleness-Learning Rate Scaling Laws for Asynchronous RLHF

Jingwei Song, Haofeng Xu, Jie Xiao +8

High-throughput RLHF systems often decouple rollout generation from policy optimization, leading to the use of stale rollouts during learner updates. In this work, we study the eff…