Parrot: Data-Driven Behavioral Priors for Reinforcement Learning

Abstract

Reinforcement learning provides a general framework for flexible decision making and control, but requires extensive data collection for each new task that an agent needs to learn. In other machine learning fields, such as natural language processing or computer vision, pre-training on large, previously collected datasets to bootstrap learning for new tasks has emerged as a powerful paradigm to reduce data requirements when learning a new task. In this paper, we ask the following question: how can we enable similarly useful pre-training for RL agents? We propose a method for pre-training behavioral priors that can capture complex input-output relationships observed in successful trials from a wide range of previously seen tasks, and we show how this learned prior can be used for rapidly learning new tasks without impeding the RL agent's ability to try out novel behaviors. We demonstrate the effectiveness of our approach in challenging robotic manipulation domains involving image observations and sparse reward functions, where our method outperforms prior works by a substantial margin.

First two authors contributed equally. Project website: https://sites.google.com/view/parrot-rl

References in corpus (21)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Auto-Encoding Variational Bayes
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Neural Discrete Representation Learning
• Soft Actor-Critic Algorithms and Applications
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• Reinforcement Learning with Deep Energy-Based Policies
• Learning to reinforcement learn
• Data-Efficient Hierarchical Reinforcement Learning
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
• Analyzing Inverse Problems with Invertible Neural Networks
• Learning and Transfer of Modulated Locomotor Controllers
• Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Prediction
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• Latent Space Policies for Hierarchical Reinforcement Learning
• Multi-Level Discovery of Deep Options
• Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation
• Improvisation through Physical Understanding: Using Novel Objects as Tools with Visual Foresight
• Continuous Relaxation of Symbolic Planner for One-Shot Imitation Learning
• Data-efficient visuomotor policy training using reinforcement learning and generative models

Cited by in corpus (5)

• Representation Matters: Offline Pretraining for Sequential Decision Making
• STAP: Sequencing Task-Agnostic Policies
• TRAIL: Near-Optimal Imitation Learning with Suboptimal Data
• Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
• Hierarchical Policies for Cluttered-Scene Grasping with Latent Plans