A-3PO: Accelerating Asynchronous LLM Training with Staleness-aware Proximal Policy Approximation

Xiaocan Li, Shiliang Wu, Zheng Shen

Decoupled PPO has been a successful reinforcement learning (RL) algorithm to deal with the high data staleness under the asynchronous RL setting. Decoupled loss used in decoupled P…