Which Mutual-Information Representation Learning Objectives are Sufficient for Control?
arXiv:2106.07278
Abstract
Mutual information maximization provides an appealing formalism for learning representations of data. In the context of reinforcement learning (RL), such representations can accelerate learning by discarding irrelevant and redundant information, while retaining the information necessary for control. Much of the prior work on these methods has addressed the practical difficulties of estimating mutual information from samples of high-dimensional observations, while comparatively less is understood about which mutual information objectives yield representations that are sufficient for RL from a theoretical perspective. In this paper, we formalize the sufficiency of a state representation for learning and representing the optimal policy, and study several popular mutual-information based objectives through this lens. Surprisingly, we find that two of these objectives can yield insufficient representations given mild and common assumptions on the structure of the MDP. We corroborate our theoretical results with empirical experiments on a simulated game environment with visual observations.
arXiv comment18 pages, 11 figures
References in corpus (27)
- Playing Atari with Deep Reinforcement Learning
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
- Learning Representations by Maximizing Mutual Information Across Views
- Learning Latent Dynamics for Planning from Pixels
- State Representation Learning for Control: An Overview
- Learning Video Representations using Contrastive Bidirectional Transformer
- Variational Intrinsic Control
- Learning to Poke by Poking: Experiential Learning of Intuitive Physics
- Near Optimal Behavior via Approximate State Abstraction
- Loss is its own Reward: Self-Supervision for Reinforcement Learning
- Contrastive Learning of Structured World Models
- DeepMDP: Learning Continuous Latent Space Models for Representation Learning
- Decoupling Representation Learning from Reinforcement Learning
- Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model
- Formal Limitations on the Measurement of Mutual Information
- Near-Optimal Representation Learning for Hierarchical Reinforcement Learning
- Hyperbolic Discounting and Learning over Multiple Horizons
- Independently Controllable Factors
- Discovery of Useful Questions as Auxiliary Tasks
- Decoupling Dynamics and Reward for Transfer Learning
- Understanding the Limitations of Variational Mutual Information Estimators
- Is a Good Representation Sufficient for Sample Efficient Reinforcement Learning?
- Comments on the Du-Kakade-Wang-Yang Lower Bounds
- Deep Reinforcement and InfoMax Learning
- Learning Visual Robotic Control Efficiently with Contrastive Pre-training and Data Augmentation
- Predictive Coding for Locally-Linear Control