324 citations · 2.8k across the 66 of their papers we have counts for
48 papers · 1 filter
Supervised Pretraining Can Learn In-Context Reinforcement Learning
Jonathan N. Lee, Annie Xie, Aldo Pacchiano +4
Large transformer models trained on diverse datasets have shown a remarkable ability to learn in-context, achieving high few-shot performance on tasks they were not explicitly trai…
Inverse Dynamics Pretraining Learns Good Representations for Multitask Imitation
David Brandfonbrener, Ofir Nachum, Joan Bruna
In recent years, domains such as natural language processing and image recognition have popularized the paradigm of using large datasets to pretrain representations that can be eff…
Multimodal Web Navigation with Instruction-Finetuned Foundation Models
Hiroki Furuta, Kuang-Huei Lee, Ofir Nachum +4
The progress of autonomous web navigation has been hindered by the dependence on billions of exploratory interactions via online reinforcement learning, and domain-specific model d…
Multi-Environment Pretraining Enables Transfer to Action Limited Datasets
David Venuto, Sherry Yang, Pieter Abbeel +3
Using massive datasets to train large-scale models has emerged as a dominant approach for broad generalization in natural language and vision applications. In reinforcement learnin…
Contrastive Value Learning: Implicit Models for Simple Offline RL
Bogdan Mazoure, Benjamin Eysenbach, Ofir Nachum +1
Model-based reinforcement learning (RL) methods are appealing in the offline setting because they allow an agent to reason about the consequences of actions without interacting wit…
Oracle Inequalities for Model Selection in Offline Reinforcement Learning
Jonathan N. Lee, George Tucker, Ofir Nachum +2
In offline reinforcement learning (RL), a learner leverages prior logged data to learn a good policy without interacting with the environment. A major challenge in applying such me…
Dichotomy of Control: Separating What You Can Control from What You Cannot
Mengjiao Yang, Dale Schuurmans, Pieter Abbeel +1
Future- or return-conditioned supervised learning is an emerging paradigm for offline reinforcement learning (RL), where the future outcome (i.e., return) associated with an observ…
Understanding HTML with Large Language Models
Izzeddin Gur, Ofir Nachum, Yingjie Miao +6
Large language models (LLMs) have shown exceptional performance on a variety of natural language tasks. Yet, their capabilities for HTML understanding -- i.e., parsing the raw HTML…
Joint Representation Training in Sequential Tasks with Shared Structure
Aldo Pacchiano, Ofir Nachum, Nilseh Tripuraneni +1
Classical theory in reinforcement learning (RL) predominantly focuses on the single task setting, where an agent learns to solve a task through trial-and-error experience, given ac…
Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters
Seyed Kamyar Seyed Ghasemipour, Shixiang Shane Gu, Ofir Nachum
Motivated by the success of ensembles for uncertainty estimation in supervised learning, we take a renewed look at how ensembles of -functions can be leveraged as the primary so…
Chain of Thought Imitation with Procedure Cloning
Mengjiao Yang, Dale Schuurmans, Pieter Abbeel +1
Imitation learning aims to extract high-performance policies from logged demonstrations of expert behavior. It is common to frame imitation learning as a supervised learning proble…
Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Error
Scott Fujimoto, David Meger, Doina Precup +2
In this work, we study the use of the Bellman equation as a surrogate objective for value prediction accuracy. While the Bellman equation is uniquely solved by the true value funct…
Model Selection in Batch Policy Optimization
Jonathan N. Lee, George Tucker, Ofir Nachum +1
We study the problem of model selection in batch policy optimization: given a fixed, partial-feedback dataset and model classes, learn a policy with performance that is competi…
Improving Zero-shot Generalization in Offline Reinforcement Learning using Generalized Similarity Functions
Bogdan Mazoure, Ilya Kostrikov, Ofir Nachum +1
Reinforcement learning (RL) agents are widely used for solving complex sequential decision making tasks, but still exhibit difficulty in generalizing to scenarios not seen during t…
TRAIL: Near-Optimal Imitation Learning with Suboptimal Data
Mengjiao Yang, Sergey Levine, Ofir Nachum
The aim in imitation learning is to learn effective policies by utilizing near-optimal expert demonstrations. However, high-quality demonstrations from human experts can be expensi…
Policy Gradients Incorporating the Future
David Venuto, Elaine Lau, Doina Precup +1
Reasoning about the future -- understanding how decisions in the present time affect outcomes in the future -- is one of the central challenges for reinforcement learning (RL), esp…
Provable Representation Learning for Imitation with Contrastive Fourier Features
Ofir Nachum, Mengjiao Yang
In imitation learning, it is common to learn a behavior policy to match an unknown target policy via max-likelihood training on a collected set of target demonstrations. In this wo…
Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization
Michael R. Zhang, Tom Le Paine, Ofir Nachum +4
Standard dynamics models for continuous control make use of feedforward computation to predict the conditional distribution of next state and reward given current state and action…
Benchmarks for Deep Off-Policy Evaluation
Justin Fu, Mohammad Norouzi, Ofir Nachum +10
Off-policy evaluation (OPE) holds the promise of being able to leverage large, offline datasets for both evaluating and selecting complex policies for decision making. The ability…
Near Optimal Policy Optimization via REPS
Aldo Pacchiano, Jonathan Lee, Peter Bartlett +1
Since its introduction a decade ago, \emph{relative entropy policy search} (REPS) has demonstrated successful policy learning on a number of simulated and real-world robotic domain…
Offline Reinforcement Learning with Fisher Divergence Critic Regularization
Ilya Kostrikov, Jonathan Tompson, Rob Fergus +1
Many modern approaches to offline Reinforcement Learning (RL) utilize behavior regularization, typically augmenting a model-free actor critic algorithm with a penalty measuring div…
Policy Information Capacity: Information-Theoretic Measure for Task Complexity in Deep Reinforcement Learning
Hiroki Furuta, Tatsuya Matsushima, Tadashi Kozuno +4
Progress in deep reinforcement learning (RL) research is largely enabled by benchmark task environments. However, analyzing the nature of those environments is often overlooked. In…
Representation Matters: Offline Pretraining for Sequential Decision Making
Mengjiao Yang, Ofir Nachum
The recent success of supervised learning methods on ever larger offline datasets has spurred interest in the reinforcement learning (RL) field to investigate whether the same para…
Offline Policy Selection under Uncertainty
Mengjiao Yang, Bo Dai, Ofir Nachum +2
The presence of uncertainty in policy evaluation significantly complicates the process of policy ranking and selection in real-world settings. We formally consider offline policy s…