10 papers · 1 filter
Reformulate LLM Reinforcement Learning for Efficient Training under Black-box Discrepancy
Jiashun Liu, Runze Liu, Xu Wan +3
Reinforcement Learning (RL) has emerged as a pivotal post-training paradigm, yet it frequently suffers from unpredictable sub-optimum performance or even training collapses. Recent…
When RL Fails after SFT: Rejuvenating Model Plasticity for Robust SFT-to-RL Handoff
Runze Liu, Jiashun Liu, Xu Wan +2
Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL) has become a standard pipeline for Large Language Model (LLM) post-training. SFT is expected to provide a usefu…
Local Guidance, Global Impact: Gaussian-Reshaped Trust Region Unlocks Behavior Transitions
Bingxu Liu, Jiashun Liu, Johan Obando-Ceron +5
While Proximal Policy Optimization (PPO) demonstrates strong performance in stationary settings, we show that its standard optimization paradigm struggles in continual and non-stat…
Learning Intractable Multimodal Policies with Reparameterization and Diversity Regularization
Ziqi Wang, Jiashun Liu, Ling Pan
Traditional continuous deep reinforcement learning (RL) algorithms employ deterministic or unimodal Gaussian actors, which cannot express complex multimodal decision distributions.…
Asymmetric Proximal Policy Optimization: mini-critics boost LLM reasoning
Jiashun Liu, Johan Obando-Ceron, Han Lu +7
Most recent RL for LLMs (RL4LLM) methods avoid explicit critics, replacing them with average advantage baselines. This shift is largely pragmatic: conventional value functions are…
Part II: ROLL Flash -- Accelerating RLVR and Agentic Training with Asynchrony
Han Lu, Zichen Liu, Shaopan Xiong +19
Synchronous Reinforcement Learning (RL) post-training has emerged as a crucial step for enhancing Large Language Models (LLMs) with diverse capabilities. However, many systems desi…