Predictive Information Accelerates Learning in RL

Abstract

The Predictive Information is the mutual information between the past and the future, I(X_past; X_future). We hypothesize that capturing the predictive information is useful in RL, since the ability to model what will happen next is necessary for success on many tasks. To test our hypothesis, we train Soft Actor-Critic (SAC) agents from pixels with an auxiliary task that learns a compressed representation of the predictive information of the RL environment dynamics using a contrastive version of the Conditional Entropy Bottleneck (CEB) objective. We refer to these as Predictive Information SAC (PI-SAC) agents. We show that PI-SAC agents can substantially improve sample efficiency over challenging baselines on tasks from the DM Control suite of continuous control environments. We evaluate PI-SAC agents by comparing against uncompressed PI-SAC agents, other compressed and uncompressed agents, and SAC agents directly trained from pixels. Our implementation is given on GitHub.

arXiv commentTo appear at NeurIPS 2020

References in corpus (19)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Representation Learning with Contrastive Predictive Coding
• Soft Actor-Critic Algorithms and Applications
• Learning deep representations by mutual information estimation and maximization
• DeepMind Control Suite
• Reinforcement Learning with Deep Energy-Based Policies
• Learning Latent Dynamics for Planning from Pixels
• Reinforcement Learning with Unsupervised Auxiliary Tasks
• Reinforcement Learning with Augmented Data
• On Variational Bounds of Mutual Information
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• The Conditional Entropy Bottleneck
• Loss is its own Reward: Self-Supervision for Reinforcement Learning
• DeepMDP: Learning Continuous Latent Space Models for Representation Learning
• Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneck
• Learners that Use Little Information
• On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models
• Predictive Information
• The Variational Bandwidth Bottleneck: Stochastic Evaluation on an Information Budget

Cited by in corpus (9)

• Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning
• Return-Based Contrastive Representation Learning for Reinforcement Learning
• Integrating Contrastive Learning with Dynamic Models for Reinforcement Learning from Images
• Multimodal Information Bottleneck for Deep Reinforcement Learning with Multiple Sensors
• PlayVirtual: Augmenting Cycle-Consistent Virtual Trajectories for Reinforcement Learning
• Test Distribution-Aware Active Learning: A Principled Approach Against Distribution Shift and Outliers
• Learning Temporally-Consistent Representations for Data-Efficient Reinforcement Learning
• Disentangled (Un)Controllable Features
• Learning Representations for Pixel-based Control: What Matters and Why?