Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning

Abstract

Off-policy reinforcement learning algorithms promise to be applicable in settings where only a fixed data-set (batch) of environment interactions is available and no new experience can be acquired. This property makes these algorithms appealing for real world problems such as robot control. In practice, however, standard off-policy algorithms fail in the batch setting for continuous control. In this paper, we propose a simple solution to this problem. It admits the use of data generated by arbitrary behavior policies and uses a learned prior -- the advantage-weighted behavior model (ABM) -- to bias the RL policy towards actions that have previously been executed and are likely to be successful on the new task. Our method can be seen as an extension of recent work on batch-RL that enables stable learning from conflicting data-sources. We find improvements on competitive baselines in a variety of RL tasks -- including standard continuous control benchmarks and multi-task learning for simulated and real-world robots.

References in corpus (12)

• Auto-Encoding Variational Bayes
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• Soft Actor-Critic Algorithms and Applications
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• DeepMind Control Suite
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• Learning Continuous Control Policies by Stochastic Value Gradients
• Learning by Playing - Solving Sparse Reward Tasks from Scratch
• Maximum a Posteriori Policy Optimisation
• Information asymmetry in KL-regularized RL
• Exploiting Hierarchy for Learning and Transfer in KL-regularized RL

Cited by in corpus (26)

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Conservative Q-Learning for Offline Reinforcement Learning
• MOPO: Model-based Offline Policy Optimization
• An Optimistic Perspective on Offline Reinforcement Learning
• COMBO: Conservative Offline Model-Based Policy Optimization
• Online and Offline Reinforcement Learning by Planning with a Learned Model
• Is Pessimism Provably Efficient for Offline RL?
• Learning Arbitrary-Goal Fabric Folding with One Hour of Real Robot Experience
• Offline Reinforcement Learning from Images with Latent Space Models
• Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning
• Multi-task Batch Reinforcement Learning with Metric Learning
• Offline Reinforcement Learning with Value-based Episodic Memory
• Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
• Skill Preferences: Learning to Extract and Execute Robotic Skills from Human Feedback
• Corruption-Robust Offline Reinforcement Learning
• Instabilities of Offline RL with Pre-Trained Neural Representation
• Offline Reinforcement Learning with Pseudometric Learning
• Offline Reinforcement Learning Hands-On
• Evaluating the progress of Deep Reinforcement Learning in the real world: aligning domain-agnostic and domain-specific research
• REPAINT: Knowledge Transfer in Deep Reinforcement Learning
• Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
• On the Optimality of Batch Policy Optimization Algorithms
• Provably Efficient Generative Adversarial Imitation Learning for Online and Offline Setting with Linear Function Approximation
• Learning Off-Policy with Online Planning
• Reinforcement Learning for Vision-based Object Manipulation with Non-parametric Policy and Action Primitives
• Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters