15 papers
Consolidating RLVR Capabilities Across Domains: A Deep Dive into Fusion Paradigms
Siye Wu, Kai Yang, Yuchen Cai +8
Reinforcement learning with verifiable rewards (RLVR) improves specific capabilities of large language models, but covering multiple capabilities often involves training separate d…
Reformulate LLM Reinforcement Learning for Efficient Training under Black-box Discrepancy
Jiashun Liu, Runze Liu, Xu Wan +3
Reinforcement Learning (RL) has emerged as a pivotal post-training paradigm, yet it frequently suffers from unpredictable sub-optimum performance or even training collapses. Recent…
When RL Fails after SFT: Rejuvenating Model Plasticity for Robust SFT-to-RL Handoff
Runze Liu, Jiashun Liu, Xu Wan +2
Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL) has become a standard pipeline for Large Language Model (LLM) post-training. SFT is expected to provide a usefu…
Local Guidance, Global Impact: Gaussian-Reshaped Trust Region Unlocks Behavior Transitions
Bingxu Liu, Jiashun Liu, Johan Obando-Ceron +5
While Proximal Policy Optimization (PPO) demonstrates strong performance in stationary settings, we show that its standard optimization paradigm struggles in continual and non-stat…
Complementary RL: Towards Efficient Experience-Driven Agent Learning
Dilxat Muhtar, Jiashun Liu, Wei Gao +8
Reinforcement Learning (RL) has emerged as a powerful paradigm for training LLM-based agents, yet remains limited by low sample efficiency, stemming not only from sparse outcome fe…
CE-RM: A Pointwise Generative Reward Model Optimized via Two-Stage Rollout and Unified Criteria
Xinyu Hu, Yancheng He, Weixun Wang +6
Automatic evaluation is crucial yet challenging for open-ended natural language generation, especially when rule-based metrics are infeasible. Compared with traditional methods, th…
Let It Flow: Agentic Crafting on Rock and Roll, Building the ROME Model within an Open Agentic Learning Ecosystem
Weixun Wang, XiaoXiao Xu, Wanhe An +86
Agentic crafting requires LLMs to operate in real-world environments over multiple turns by taking actions, observing outcomes, and iteratively refining artifacts. Despite its impo…
Learning Intractable Multimodal Policies with Reparameterization and Diversity Regularization
Ziqi Wang, Jiashun Liu, Ling Pan
Traditional continuous deep reinforcement learning (RL) algorithms employ deterministic or unimodal Gaussian actors, which cannot express complex multimodal decision distributions.…
Asymmetric Proximal Policy Optimization: mini-critics boost LLM reasoning
Jiashun Liu, Johan Obando-Ceron, Han Lu +7
Most recent RL for LLMs (RL4LLM) methods avoid explicit critics, replacing them with average advantage baselines. This shift is largely pragmatic: conventional value functions are…
Part II: ROLL Flash -- Accelerating RLVR and Agentic Training with Asynchrony
Han Lu, Zichen Liu, Shaopan Xiong +19
Synchronous Reinforcement Learning (RL) post-training has emerged as a crucial step for enhancing Large Language Models (LLMs) with diverse capabilities. However, many systems desi…
Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization
Yang Li, Zhichen Dong, Yuhan Sun +9
The reasoning pattern of Large language models (LLMs) remains opaque, and reinforcement learning (RL) typically applies uniform credit across an entire generation, blurring the dis…
Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning
Zihe Liu, Jiashun Liu, Yancheng He +13
Reinforcement learning for LLM reasoning has rapidly emerged as a prominent research area, marked by a significant surge in related studies on both algorithmic innovations and prac…
The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning
Jiashun Liu, Johan Obando-Ceron, Pablo Samuel Castro +2
Off-policy deep reinforcement learning (RL) typically leverages replay buffers for reusing past experiences during learning. This can help improve sample efficiency when the collec…
Measure gradients, not activations! Enhancing neuronal activity in deep reinforcement learning
Jiashun Liu, Zihao Wu, Johan Obando-Ceron +3
Deep reinforcement learning (RL) agents frequently suffer from neuronal activity loss, which impairs their ability to adapt to new data and learn continually. A common method to qu…
Neuroplastic Expansion in Deep Reinforcement Learning
Jiashun Liu, Johan Obando-Ceron, Aaron Courville +1
The loss of plasticity in learning agents, analogous to the solidification of neural pathways in biological brains, significantly impedes learning and adaptation in reinforcement l…