61 citations · 252 across the 80 of their papers we have counts for
51 papers · 1 filter
Variance-Optimal State Resampling for Reinforcement Learning with Verifiable Rewards
Yibo Wang, Yuanxin Liu, Zhaoran Wang
Reinforcement learning with verifiable rewards spends much of its rollout budget on states that yield no learning signal. We recast this as a question about the sampling distributi…
Parallax: Parameterized Local Linear Attention for Language Modeling
Yifei Zuo, Dhruv Pai, Zhichen Zeng +3
Large Language Models (LLMs) have become the central paradigm in artificial intelligence, yet the core computational primitive of attention has remained structurally unchanged. Loc…
HiPER: Hierarchical Reinforcement Learning with Explicit Credit Assignment for Large Language Model Agents
Jiangweizhi Peng, Yuanxin Liu, Ruida Zhou +4
Training LLMs as interactive agents for multi-turn decision-making remains challenging, particularly in long-horizon tasks with sparse and delayed rewards, where agents must execut…
Learning to Reason as Action Abstractions with Scalable Mid-Training RL
Shenao Zhang, Donghan Yu, Yihao Feng +4
Large language models excel with reinforcement learning (RL), but fully unlocking this potential requires a mid-training stage. An effective mid-training phase should identify a co…
Beyond Markovian: Reflective Exploration via Bayes-Adaptive RL for LLM Reasoning
Shenao Zhang, Yaqing Wang, Yinxiao Liu +5
Large Language Models (LLMs) trained via Reinforcement Learning (RL) have exhibited strong reasoning capabilities and emergent reflective behaviors, such as rethinking and error co…
BRiTE: Bootstrapping Reinforced Thinking Process to Enhance Language Model Reasoning
Han Zhong, Yutong Yin, Shenao Zhang +10
Large Language Models (LLMs) have demonstrated remarkable capabilities in complex reasoning tasks, yet generating reliable reasoning processes remains a significant challenge. We p…
Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
Shenao Zhang, Zhihan Liu, Boyi Liu +6
Preference alignment in Large Language Models (LLMs) has significantly improved their ability to adhere to human instructions and intentions. However, existing direct alignment alg…
Self-Exploring Language Models: Active Preference Elicitation for Online Alignment
Shenao Zhang, Donghan Yu, Hiteshi Sharma +6
Preference optimization, particularly through Reinforcement Learning from Human Feedback (RLHF), has achieved significant success in aligning Large Language Models (LLMs) to adhere…
Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer
Zhihan Liu, Miao Lu, Shenao Zhang +5
Aligning generative models with human preference via RLHF typically suffers from overoptimization, where an imperfectly learned reward model can misguide the generative model to ou…
A Mean-Field Analysis of Neural Stochastic Gradient Descent-Ascent for Functional Minimax Optimization
Yuchen Zhu, Yufeng Zhang, Zhaoran Wang +2
This paper studies minimax optimization problems defined over infinite-dimensional function classes of overparameterized two-layer neural networks. In particular, we consider the m…
How Can LLM Guide RL? A Value-Based Approach
Shenao Zhang, Sirui Zheng, Shuqi Ke +6
Reinforcement learning (RL) has become the de facto standard practice for sequential decision-making problems by improving future acting policies with feedback. However, RL algorit…
Double Duality: Variational Primal-Dual Policy Optimization for Constrained Reinforcement Learning
Zihao Li, Boyi Liu, Zhuoran Yang +2
We study the Constrained Convex Markov Decision Process (MDP), where the goal is to minimize a convex functional of the visitation measure, subject to a convex constraint. Designin…
Model-Based Reparameterization Policy Gradient Methods: Theory and Practical Algorithms
Shenao Zhang, Boyi Liu, Zhaoran Wang +1
ReParameterization (RP) Policy Gradient Methods (PGMs) have been widely adopted for continuous control tasks in robotics and computer graphics. However, recent studies have reveale…
Posterior Sampling for Competitive RL: Function Approximation and Partial Observation
Shuang Qiu, Ziyu Dai, Han Zhong +3
This paper investigates posterior sampling algorithms for competitive reinforcement learning (RL) in the context of general function approximations. Focusing on zero-sum Markov gam…
Sample-Efficient Multi-Agent RL: An Optimization Perspective
Nuoya Xiong, Zhihan Liu, Zhaoran Wang +1
We study multi-agent reinforcement learning (MARL) for the general-sum Markov Games (MGs) under the general function approximation. In order to find the minimum assumption for samp…
A General Framework for Sequential Decision-Making under Adaptivity Constraints
Nuoya Xiong, Zhaoran Wang, Zhuoran Yang
We take the first step in studying general sequential decision-making under two adaptivity constraints: rare policy switch and batch learning. First, we provide a general class cal…
Provably Efficient Generalized Lagrangian Policy Optimization for Safe Multi-Agent Reinforcement Learning
Dongsheng Ding, Xiaohan Wei, Zhuoran Yang +2
We examine online safe multi-agent reinforcement learning using constrained Markov games in which agents compete by maximizing their expected total rewards under a constraint on ex…
Local Optimization Achieves Global Optimality in Multi-Agent Reinforcement Learning
Yulai Zhao, Zhuoran Yang, Zhaoran Wang +1
Policy optimization methods with function approximation are widely used in multi-agent reinforcement learning. However, it remains elusive how to design such algorithms with statis…
Maximize to Explore: One Objective Function Fusing Estimation, Planning, and Exploration
Zhihan Liu, Miao Lu, Wei Xiong +6
In online reinforcement learning (online RL), balancing exploration and exploitation is crucial for finding an optimal policy in a sample-efficient way. To achieve this, existing s…
Dynamic Datasets and Market Environments for Financial Reinforcement Learning
Xiao-Yang Liu, Ziyi Xia, Hongyang Yang +6
The financial market is a particularly challenging playground for deep reinforcement learning due to its unique feature of dynamic datasets. Building high-quality market environmen…
Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization
Haoran Xu, Li Jiang, Jianxiong Li +4
Most offline reinforcement learning (RL) methods suffer from the trade-off between improving the policy to surpass the behavior policy and constraining the policy to limit the devi…
A Unified Framework of Policy Learning for Contextual Bandit with Confounding Bias and Missing Observations
Siyu Chen, Yitan Wang, Zhaoran Wang +1
We study the offline contextual bandit problem, where we aim to acquire an optimal policy using observational data. However, this data usually contains two deficiencies: (i) some v…
Offline Policy Optimization in RL with Variance Regularizaton
Riashat Islam, Samarth Sinha, Homanga Bharadhwaj +6
Learning policies from fixed offline datasets is a key challenge to scale up reinforcement learning (RL) algorithms towards practical applications. This is often because off-policy…
An Analysis of Attention via the Lens of Exchangeability and Latent Variable Models
Yufeng Zhang, Boyi Liu, Qi Cai +2
With the attention mechanism, transformers achieve significant empirical successes. Despite the intuitive understanding that transformers perform relational inference over long seq…