7 papers
When Is Enough Enough in Self-Evolving LLM Systems?
Enoch Yin, Bin Liu, Zhengling Qi
Self-evolving large language model (LLM) systems repeatedly propose, evaluate, and incorporate updates to prompts, skills, or other persistent artifacts. Despite their growing effe…
Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation
Wenbo Zhang, Wenzhuo Zhou, Hengrui Cai +1
Direct preference optimization DPO is a promising offline approach for aligning large language models (LLMs) due to its simplicity, computational efficiency, and implicit modeling…
OPPO: Bayesian Value Recursion for Token-Level Credit Assignment in LLM Reasoning
Yu Li, Rui Miao, Tian Lan +1
Reinforcement learning with verifiable rewards has become the standard recipe for improving LLM reasoning, but the dominant algorithm GRPO assigns a single trajectory-level advanta…
ARISE: Agent Reasoning with Intrinsic Skill Evolution in Hierarchical Reinforcement Learning
Yu Li, Rui Miao, Zhengling Qi +1
The dominant paradigm for improving mathematical reasoning in language models relies on Reinforcement Learning with verifiable rewards. Yet existing methods treat each problem inst…
A Principled Path to Fitted Distributional Evaluation
Sungee Hong, Jiayi Wang, Zhengling Qi +1
In reinforcement learning, distributional off-policy evaluation (OPE) focuses on estimating the return distribution of a target policy using offline data collected under a differen…
A Fine-grained Analysis of Fitted Q-evaluation: Beyond Parametric Models
Jiayi Wang, Zhengling Qi, Raymond K. W. Wong
In this paper, we delve into the statistical analysis of the fitted Q-evaluation (FQE) method, which focuses on estimating the value of a target policy using offline data generated…
Distributional Off-policy Evaluation with Bellman Residual Minimization
Sungee Hong, Zhengling Qi, Raymond K. W. Wong
We study distributional off-policy evaluation (OPE), of which the goal is to learn the distribution of the return for a target policy using offline data generated by a different po…