67 citations · 239 across the 26 of their papers we have counts for
26 papers
Offline Reinforcement Learning in Large State Spaces: Algorithms and Guarantees
Nan Jiang, Tengyang Xie
This article introduces the theory of offline reinforcement learning in large state spaces, where good policies are learned from historical data without online interactions with th…
Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization
Audrey Huang, Wenhao Zhan, Tengyang Xie +4
Language model alignment methods such as reinforcement learning from human feedback (RLHF) have led to impressive advances in language model capabilities, but are limited by a wide…
Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts
Haoxiang Wang, Wei Xiong, Tengyang Xie +2
Reinforcement learning from human feedback (RLHF) has emerged as the primary method for aligning large language models (LLMs) with human preferences. The RLHF process typically sta…
Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Models
Xiang Ji, Sanjeev Kulkarni, Mengdi Wang +1
This work studies the challenge of aligning large language models (LLMs) with offline preference data. We focus on alignment by Reinforcement Learning from Human Feedback (RLHF) in…
Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data
Fahim Tajwar, Anikait Singh, Archit Sharma +6
Learning from preference labels plays a crucial role in fine-tuning large language models. There are several distinct approaches for preference fine-tuning, including supervised le…
Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF
Tengyang Xie, Dylan J. Foster, Akshay Krishnamurthy +3
Reinforcement learning from human feedback (RLHF) has emerged as a central tool for language model alignment. We consider online exploration in RLHF, which exploits interactive acc…
Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences
Corby Rosset, Ching-An Cheng, Arindam Mitra +3
This paper studies post-training large language models (LLMs) using preference feedback from a powerful oracle to help a model iteratively improve over itself. The typical approach…
Towards Principled Representation Learning from Videos for Reinforcement Learning
Dipendra Misra, Akanksha Saran, Tengyang Xie +2
We study pre-training representations for decision-making using video data, which is abundantly available for tasks such as game agents and software testing. Even though significan…
CounterCurate: Enhancing Physical and Semantic Visio-Linguistic Compositional Reasoning via Counterfactual Examples
Jianrui Zhang, Mu Cai, Tengyang Xie +1
We propose CounterCurate, a framework to comprehensively improve the visio-linguistic compositional reasoning capability for both contrastive and generative multimodal models. In p…
Harnessing Density Ratios for Online Reinforcement Learning
Philip Amortila, Dylan J. Foster, Nan Jiang +2
The theories of offline and online reinforcement learning, despite having evolved in parallel, have begun to show signs of the possibility for a unification, with algorithms and an…
Adversarial Model for Offline Reinforcement Learning
Mohak Bhardwaj, Tengyang Xie, Byron Boots +2
We propose a novel model-based offline Reinforcement Learning (RL) framework, called Adversarial Model for Offline Reinforcement Learning (ARMOR), which can robustly learn policies…
ARMOR: A Model-based Framework for Improving Arbitrary Baseline Policies with Offline Data
Tengyang Xie, Mohak Bhardwaj, Nan Jiang +1
We propose a new model-based offline RL framework, called Adversarial Models for Offline Reinforcement Learning (ARMOR), which can robustly learn policies to improve upon an arbitr…
The Role of Coverage in Online Reinforcement Learning
Tengyang Xie, Dylan J. Foster, Yu Bai +2
Coverage conditions -- which assert that the data logging distribution adequately covers the state space -- play a fundamental role in determining the sample complexity of offline…
Interaction-Grounded Learning with Action-inclusive Feedback
Tengyang Xie, Akanksha Saran, Dylan J. Foster +5
Consider the problem setting of Interaction-Grounded Learning (IGL), in which a learner's goal is to optimally interact with the environment with no explicit reward to ground its p…
Adversarially Trained Actor Critic for Offline Reinforcement Learning
Ching-An Cheng, Tengyang Xie, Nan Jiang +1
We propose Adversarially Trained Actor Critic (ATAC), a new model-free algorithm for offline reinforcement learning (RL) under insufficient data coverage, based on the concept of r…
Interaction-Grounded Learning
Tengyang Xie, John Langford, Paul Mineiro +1
Consider a prosthetic arm, learning to adapt to its user's control signals. We propose Interaction-Grounded Learning for this novel setting, in which a learner's goal is to interac…
Bellman-consistent Pessimism for Offline Reinforcement Learning
Tengyang Xie, Ching-An Cheng, Nan Jiang +2
The use of pessimism, when reasoning about datasets lacking exhaustive exploration has recently gained prominence in offline reinforcement learning. Despite the robustness it adds…
Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning
Tengyang Xie, Nan Jiang, Huan Wang +2
Recent theoretical work studies sample-efficient reinforcement learning (RL) extensively in two settings: learning interactively in the environment (online RL), or learning from an…
Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiency
Masatoshi Uehara, Masaaki Imaizumi, Nan Jiang +3
We offer a theoretical characterization of off-policy evaluation (OPE) in reinforcement learning using function approximation for marginal importance weights and -functions when…
A Variant of the Wang-Foster-Kakade Lower Bound for the Discounted Setting
Philip Amortila, Nan Jiang, Tengyang Xie
Recently, Wang et al. (2020) showed a highly intriguing hardness result for batch reinforcement learning (RL) with linearly realizable value function and good feature coverage in t…
Batch Value-function Approximation with Only Realizability
Tengyang Xie, Nan Jiang
We make progress in a long-standing problem of batch reinforcement learning (RL): learning from an exploratory and polynomial-sized dataset, using a realizable and otherw…
Q* Approximation Schemes for Batch Reinforcement Learning: A Theoretical Comparison
Tengyang Xie, Nan Jiang
We prove performance guarantees of two algorithms for approximating in batch reinforcement learning. Compared to classical iterative methods such as Fitted Q-Iteration---…
Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling
Tengyang Xie, Yifei Ma, Yu-Xiang Wang
Motivated by the many real-world applications of reinforcement learning (RL) that require safe-policy iterations, we consider the problem of off-policy evaluation (OPE) -- the prob…
Provably Efficient Q-Learning with Low Switching Cost
Yu Bai, Tengyang Xie, Nan Jiang +1
We take initial steps in studying PAC-MDP algorithms with limited adaptivity, that is, algorithms that change its exploration policy as infrequently as possible during regret minim…