20 papers
Diagnosing On-Policy Self-Distillation for Reasoning Language Models
Yang Li, Gongle Xue, Yuheng Yuan +4
On-policy self-distillation (OPSD) has attracted growing interest as a promising approach to improve the reasoning ability of language models. Without external rewards nor a separa…
Understanding the Synergy between SFT, RLVR, and OPD in LLM Post-Training
Emre Can Acikgoz, Yang Li, Zeyu Leo Liu +4
Modern LLM post-training composes supervised fine-tuning (SFT), reinforcement learning with verifiable rewards (RLVR), and on-policy distillation (OPD) into multi-stage pipelines,…
Just-in-Time Memory: Learning to Curate Task-Adaptive Memory for LLM Agents
Yefan Zhou, Yang Li, Zeyu Leo Liu +2
Agentic memory systems reuse past experience to improve future performance, yet most existing designs curate memory at write time: once a task is completed, its trajectory is disti…
EvoHarnessBench: Can Your Agents Keep Pace with an Evolving Harness?
Zixuan Ke, Vaidehi Patil, Haizhou Shi +9
Modern LLM-based agents operate through a harness of tools, reusable skills, and specialist agents that shapes what they observe and what they can do. In practice, this harness con…
Procedural Memory Distillation: Online Reflection for Self-Improving Language Models
Ye Liu, Srijan Bansal, Bo Pang +6
Reinforcement learning with verifiable rewards (RLVR), along with recent selfdistillation variants such as SDPO, evaluates each rollout against a verifier and updates the policy fr…
CAST: Non-Privileged Clipped Asymmetric Self-Teaching with Advantage Flipping for GRPO
Yang Li, Gongle Xue, Yijia Guo +3
Reinforcement learning with verifiable rewards (RLVR), especially Group Relative Policy Optimization (GRPO), has been widely used to improve reasoning in large language models. How…
Efficient Agentic Reinforcement Learning with On-Policy Intrinsic Knowledge Boundary Enhancement
Dingwei Chen, Zefang Zong, Zhipeng Ma +5
Agentic reinforcement learning (RL) has proven effective for training LLM-based agents with external tool-use capabilities. However, we identify that agentic RL training induces in…
When Search Becomes Memory: Accelerating Robot Design Discovery with Self-Evolving Skills
Yunfei Wang, Xiaohao Xu, Yang Li +1
Large language models (LLMs) are increasingly used as proposal generators for evolutionary robot design, yet most loops remain memoryless: simulator results shape the next populati…
Beyond Mode Collapse: Distribution Matching for Diverse Reasoning
Xiaozhe Li, Yang Li, Xinyu Fang +10
On-policy reinforcement learning methods like GRPO suffer from mode collapse: they exhibit reduced solution diversity, concentrating probability mass on a single solution once disc…
What and When to Distill: Selective Hindsight Distillation for Multi-Turn Agents
Xiaozhe Li, Tianyi Lyu, Yang Li +6
Reinforcement learning can train LLM agents from sparse task rewards, but long-horizon credit assignment remains challenging: a single success-or-failure signal must be distributed…
TabQL: In-Context Q-Learning with Tabular Foundation Models
Qisai Liu, Zhanhong Jiang, Timilehin Ayanlade +4
We propose Tabular Q-Learning (TabQL), a reinforcement learning framework that replaces the conventional parametric Q-network in Deep Q-Learning (DQN) with a tabular foundation mod…
Learning from Language Feedback via Variational Policy Distillation
Yang Li, Erik Nijkamp, Semih Yavuz +1
Reinforcement learning from verifiable rewards (RLVR) suffers from sparse outcome signals, creating severe exploration bottlenecks on complex reasoning tasks. Recent on-policy self…
LACE: Lattice Attention for Cross-thread Exploration
Yang Li, Zirui Zhang, Yang Liu +1
Current large language models reason in isolation. Although it is common to sample multiple reasoning paths in parallel, these trajectories do not interact, and often fail in the s…
Forge: Quality-Aware Reinforcement Learning for NP-Hard Optimization in LLMs
Xiaozhe Li, Xinyu Fang, Shengyuan Ding +5
Large Language Models (LLMs) have achieved remarkable success on reasoning benchmarks through Reinforcement Learning with Verifiable Rewards (RLVR), excelling at tasks such as math…
Escaping the Context Bottleneck: Active Context Curation for LLM Agents via Reinforcement Learning
Xiaozhe Li, Tianyi Lyu, Yizhao Yang +6
Large Language Models (LLMs) struggle with long-horizon tasks due to the "context bottleneck" and the "lost-in-the-middle" phenomenon, where accumulated noise from verbose environm…
Beyond Scaling: Assessing Strategic Reasoning and Rapid Decision-Making Capability of LLMs in Zero-sum Environments
Yang Li, Xing Chen, Yutao Liu +5
Large Language Models (LLMs) have achieved strong performance on static reasoning benchmarks, yet their effectiveness as interactive agents operating in adversarial, time-sensitive…
Reasoning Palette: Modulating Reasoning via Latent Contextualization for Controllable Exploration for (V)LMs
Rujiao Long, Yang Li, Xingyao Zhang +7
Exploration capacity shapes both inference-time performance and reinforcement learning (RL) training for large (vision-) language models, as stochastic sampling often yields redund…
Let It Flow: Agentic Crafting on Rock and Roll, Building the ROME Model within an Open Agentic Learning Ecosystem
Weixun Wang, XiaoXiao Xu, Wanhe An +86
Agentic crafting requires LLMs to operate in real-world environments over multiple turns by taking actions, observing outcomes, and iteratively refining artifacts. Despite its impo…
Part II: ROLL Flash -- Accelerating RLVR and Agentic Training with Asynchrony
Han Lu, Zichen Liu, Shaopan Xiong +19
Synchronous Reinforcement Learning (RL) post-training has emerged as a crucial step for enhancing Large Language Models (LLMs) with diverse capabilities. However, many systems desi…
Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization
Yang Li, Zhichen Dong, Yuhan Sun +9
The reasoning pattern of Large language models (LLMs) remains opaque, and reinforcement learning (RL) typically applies uniform credit across an entire generation, blurring the dis…