Representation Matters: Offline Pretraining for Sequential Decision Making

Abstract

The recent success of supervised learning methods on ever larger offline datasets has spurred interest in the reinforcement learning (RL) field to investigate whether the same paradigms can be translated to RL algorithms. This research area, known as offline RL, has largely focused on offline policy optimization, aiming to find a return-maximizing policy exclusively from offline data. In this paper, we consider a slightly different approach to incorporating offline data into sequential decision-making. We aim to answer the question, what unsupervised objectives applied to offline datasets are able to learn state representations which elevate performance on downstream tasks, whether those downstream tasks be online RL, imitation learning from expert demonstrations, or even offline policy optimization based on the same offline dataset? Through a variety of experiments utilizing standard offline RL datasets, we find that the use of pretraining with unsupervised learning objectives can dramatically improve the performance of policy learning algorithms that otherwise yield mediocre performance on their own. Extensive ablations further provide insights into what components of these unsupervised objectives -- e.g., reward prediction, continuous or discrete representations, pretraining or finetuning -- are most important and in which settings.

References in corpus (23)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Language Models are Few-Shot Learners
• Soft Actor-Critic Algorithms and Applications
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Searching for Activation Functions
• Conservative Q-Learning for Offline Reinforcement Learning
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Challenges of Real-World Reinforcement Learning
• Behavior Regularized Offline Reinforcement Learning
• MOPO: Model-based Offline Policy Optimization
• Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• Dream to Control: Learning Behaviors by Latent Imagination
• Learning Invariant Representations for Reinforcement Learning without Reconstruction
• Decoupling Representation Learning from Reinforcement Learning
• Near-Optimal Representation Learning for Hierarchical Reinforcement Learning
• Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization
• Value Prediction Network
• Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning
• Parrot: Data-Driven Behavioral Priors for Reinforcement Learning
• OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning
• Prediction, Consistency, Curvature: Representation Learning for Locally-Linear Control
• The Laplacian in RL: Learning Representations with Efficient Approximations

Cited by in corpus (8)

• What Matters in Learning from Offline Human Demonstrations for Robot Manipulation
• Provable Representation Learning for Imitation with Contrastive Fourier Features
• Corruption-Robust Offline Reinforcement Learning
• Cross-Trajectory Representation Learning for Zero-Shot Generalization in RL
• Representation Learning for Online and Offline RL in Low-rank MDPs
• Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
• Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulation
• Policy Gradients Incorporating the Future