Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learning

Abstract

Learning a good representation is an essential component for deep reinforcement learning (RL). Representation learning is especially important in multitask and partially observable settings where building a representation of the unknown environment is crucial to solve the tasks. Here we introduce Prediction of Bootstrap Latents (PBL), a simple and flexible self-supervised representation learning algorithm for multitask deep RL. PBL builds on multistep predictive representations of future observations, and focuses on capturing structured information about environment dynamics. Specifically, PBL trains its representation by predicting latent embeddings of future observations. These latent embeddings are themselves trained to be predictive of the aforementioned representations. These predictions form a bootstrapping effect, allowing the agent to learn more about the key aspects of the environment dynamics. In addition, by defining prediction tasks completely in latent space, PBL provides the flexibility of using multimodal observations involving pixel images, language instructions, rewards and more. We show in our experiments that PBL delivers across-the-board improved performance over state of the art deep RL agents in the DMLab-30 and Atari-57 multitask setting.

References in corpus (13)

• Auto-Encoding Variational Bayes
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• DeepMind Control Suite
• VIME: Variational Information Maximizing Exploration
• Recurrent World Models Facilitate Policy Evolution
• Learning to Navigate in Complex Environments
• Reinforcement Learning with Unsupervised Auxiliary Tasks
• SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning
• Never Give Up: Learning Directed Exploration Strategies
• Sample Complexity of Multi-task Reinforcement Learning
• Value Prediction Network
• Neural Predictive Belief Representations
• World Discovery Models

Cited by in corpus (15)

• Decoupling Representation Learning from Reinforcement Learning
• Data-Efficient Reinforcement Learning with Self-Predictive Representations
• Auxiliary-task Based Deep Reinforcement Learning for Participant Selection Problem in Mobile Crowdsourcing
• Return-Based Contrastive Representation Learning for Reinforcement Learning
• Mine Your Own vieW: Self-Supervised Learning Through Across-Sample Prediction
• Muesli: Combining Improvements in Policy Optimization
• Drop, Swap, and Generate: A Self-Supervised Approach for Generating Neural Activity
• PlayVirtual: Augmenting Cycle-Consistent Virtual Trajectories for Reinforcement Learning
• Bootstrap Equilibrium and Probabilistic Speaker Representation Learning for Self-supervised Speaker Verification
• Self-supervised Graph Representation Learning via Bootstrapping
• Sequential Topological Representations for Predictive Models of Deformable Objects
• DS-Net++: Dynamic Weight Slicing for Efficient Inference in CNNs and Transformers
• Pretrained Encoders are All You Need
• Learning State Representations from Random Deep Action-conditional Predictions
• Effects of sampling and horizon in predictive reinforcement learning