BRACE: Anchored Bellman-Residual Correction for Stale Critics in Asynchronous RL

Guanqun Zhao, Zijun Xie, Binbin Zheng +3

Asynchronous reinforcement learning has become the standard way to scale training for large language models (LLM), but the resulting policy lag biases the critic toward the stale b…