382 citations · 978 across the 50 of their papers we have counts for
64 papers
Tail-Likelihood Reinforcement Learning
Shrinivas Ramasubramanian, Daman Arora, Fahim Tajwar +11
Reinforcement learning typically optimizes average reward. For generative policies, the average can hide an important distinction: two policies can achieve the same mean reward whi…
Expanding the Capabilities of Reinforcement Learning via Text Feedback
Yuda Song, Lili Chen, Fahim Tajwar +5
The success of RL for LLM post-training stems from an unreasonably uninformative source: a single bit of information per rollout as binary reward or preference label. At the other…
To Distill or Decide? Understanding the Algorithmic Trade-off in Partially Observable Reinforcement Learning
Yuda Song, Dhruv Rohatgi, Aarti Singh +1
Partial observability is a notorious challenge in reinforcement learning (RL), due to the need to learn complex, history-dependent policies. Recent empirical successes have used pr…
All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuning
Gokul Swamy, Sanjiban Choudhury, Wen Sun +2
From a first-principles perspective, it may seem odd that the strongest results in foundation model fine-tuning (FT) are achieved via a relatively complex, two-stage training proce…
The Importance of Online Data: Understanding Preference Fine-tuning via Coverage
Yuda Song, Gokul Swamy, Aarti Singh +2
Learning from human preference data has emerged as the dominant paradigm for fine-tuning large language models (LLMs). The two most common families of techniques -- online reinforc…
Hybrid Reinforcement Learning from Offline Observation Alone
Yuda Song, J. Andrew Bagnell, Aarti Singh
We consider the hybrid reinforcement learning setting where the agent has access to both offline data and online interactive access. While Reinforcement Learning (RL) research typi…
REBEL: Reinforcement Learning via Regressing Relative Rewards
Zhaolin Gao, Jonathan D. Chang, Wenhao Zhan +7
While originally developed for continuous control problems, Proximal Policy Optimization (PPO) has emerged as the work-horse of a variety of reinforcement learning (RL) application…
The Virtues of Pessimism in Inverse Reinforcement Learning
David Wu, Gokul Swamy, J. Andrew Bagnell +2
Inverse Reinforcement Learning (IRL) is a powerful framework for learning complex behaviors from expert demonstrations. However, it traditionally requires repeatedly solving a comp…
Hybrid Inverse Reinforcement Learning
Juntao Ren, Gokul Swamy, Zhiwei Steven Wu +2
The inverse reinforcement learning approach to imitation learning is a double-edged sword. On the one hand, it can enable learning from a smaller number of expert demonstrations wi…
Inverse Reinforcement Learning without Reinforcement Learning
Gokul Swamy, Sanjiban Choudhury, J. Andrew Bagnell +1
Inverse Reinforcement Learning (IRL) is a powerful set of techniques for imitation learning that aims to learn a reward function that rationalizes expert demonstrations. Unfortunat…
The Virtues of Laziness in Model-based RL: A Unified Objective and Algorithms
Anirudh Vemula, Yuda Song, Aarti Singh +2
We propose a novel approach to addressing two fundamental challenges in Model-based Reinforcement Learning (MBRL): the computational expense of repeatedly finding a good policy in…
Hybrid RL: Using Both Offline and Online Data Can Make RL Efficient
Yuda Song, Yifei Zhou, Ayush Sekhari +3
We consider a hybrid reinforcement learning setting (Hybrid RL), in which an agent has access to an offline dataset and the ability to collect experience via real-world online inte…
Game-Theoretic Algorithms for Conditional Moment Matching
Gokul Swamy, Sanjiban Choudhury, J. Andrew Bagnell +1
A variety of problems in econometrics and machine learning, including instrumental variable regression and Bellman residual minimization, can be formulated as satisfying a set of c…
Sequence Model Imitation Learning with Unobserved Contexts
Gokul Swamy, Sanjiban Choudhury, J. Andrew Bagnell +1
We consider imitation learning problems where the learner's ability to mimic the expert increases throughout the course of an episode as more information is revealed. One example o…
Minimax Optimal Online Imitation Learning via Replay Estimation
Gokul Swamy, Nived Rajaraman, Matthew Peng +5
Online imitation learning is the problem of how best to mimic expert demonstrations, given access to the environment or an accurate simulator. Prior work has shown that in the infi…
Causal Imitation Learning under Temporally Correlated Noise
Gokul Swamy, Sanjiban Choudhury, J. Andrew Bagnell +1
We develop algorithms for imitation learning from policy data that was corrupted by temporally correlated noise in expert actions. When noise affects multiple timesteps of recorded…
A Critique of Strictly Batch Imitation Learning
Gokul Swamy, Sanjiban Choudhury, J. Andrew Bagnell +1
Recent work by Jarrett et al. attempts to frame the problem of offline imitation learning (IL) as one of learning a joint energy-based model, with the hope of out-performing standa…
Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gap
Gokul Swamy, Sanjiban Choudhury, J. Andrew Bagnell +1
We provide a unifying view of a large family of previous imitation learning algorithms through the lens of moment matching. At its core, our classification scheme is based on wheth…
Feedback in Imitation Learning: The Three Regimes of Covariate Shift
Jonathan Spencer, Sanjiban Choudhury, Arun Venkatraman +2
Imitation learning practitioners have often noted that conditioning policies on previous actions leads to a dramatic divergence between "held out" error and performance of the lear…
CMAX++ : Leveraging Experience in Planning and Execution using Inaccurate Models
Anirudh Vemula, J. Andrew Bagnell, Maxim Likhachev
Given access to accurate dynamical models, modern planning approaches are effective in computing feasible and optimal plans for repetitive robotic tasks. However, it is difficult t…
TRON: A Fast Solver for Trajectory Optimization with Non-Smooth Cost Functions
Anirudh Vemula, J. Andrew Bagnell
Trajectory optimization is an important tool for control and planning of complex, underactuated robots, and has shown impressive results in real world robotic tasks. However, in ap…
Planning and Execution using Inaccurate Models with Provable Guarantees
Anirudh Vemula, Yash Oza, J. Andrew Bagnell +1
Models used in modern planning problems to simulate outcomes of real world action executions are becoming increasingly complex, ranging from simulators that do physics-based reason…
Provably Efficient Imitation Learning from Observation Alone
Wen Sun, Anirudh Vemula, Byron Boots +1
We study Imitation Learning (IL) from Observations alone (ILFO) in large-scale MDPs. While most IL algorithms rely on an expert to directly provide actions to the learner, in this…
Contrasting Exploration in Parameter and Action Space: A Zeroth-Order Optimization Perspective
Anirudh Vemula, Wen Sun, J. Andrew Bagnell
Black-box optimizers that explore in parameter space have often been shown to outperform more sophisticated action space exploration methods developed specifically for the reinforc…