InfoBot: Transfer and Exploration via the Information Bottleneck
arXiv:1901.10902
Abstract
A central challenge in reinforcement learning is discovering effective policies for tasks where rewards are sparsely distributed. We postulate that in the absence of useful reward signals, an effective exploration strategy should seek out {\it decision states}. These states lie at critical junctions in the state space from where the agent can transition to new, potentially unexplored regions. We propose to learn about decision states from prior experience. By training a goal-conditioned policy with an information bottleneck, we can identify decision states by examining where the model actually leverages the goal state. We find that this simple mechanism effectively identifies decision states, even in partially observed settings. In effect, the model learns the sensory cues that correlate with potential subgoals. In new environments, this model can then identify novel subgoals for further exploration, guiding the agent through a sequence of potential decision states and through new regions of the state space.
arXiv commentAccepted at ICLR'19
References in corpus (16)
- Auto-Encoding Variational Bayes
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- The information bottleneck method
- Equitability, mutual information, and the maximal information coefficient
- VIME: Variational Information Maximizing Exploration
- Count-Based Exploration with Neural Density Models
- Unifying Count-Based Exploration and Intrinsic Motivation
- Imagination-Augmented Agents for Deep Reinforcement Learning
- Emergence of Grounded Compositional Language in Multi-Agent Populations
- Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research
- Distral: Robust Multitask Reinforcement Learning
- Meta-Reinforcement Learning of Structured Exploration Strategies
- Nonlinear Information Bottleneck
- Relevant sparse codes with variational information bottleneck
- Learning to Share and Hide Intentions using Information Regularization
- Recall Traces: Backtracking Models for Efficient Reinforcement Learning
Cited by in corpus (19)
- Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representations
- Revisiting Training Strategies and Generalization Performance in Deep Metric Learning
- Deep reinforcement learning with a particle dynamics environment applied to emergency evacuation of a room with obstacles
- CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning
- BeBold: Exploration Beyond the Boundary of Explored Regions
- Invariant Policy Optimization: Towards Stronger Generalization in Reinforcement Learning
- Diversity inducing Information Bottleneck in Model Ensembles
- SOAC: The Soft Option Actor-Critic Architecture
- Recognizing Predictive Substructures with Subgraph Information Bottleneck
- Weakly-Supervised Reinforcement Learning for Controllable Behavior
- IR-VIC: Unsupervised Discovery of Sub-goals for Transfer in RL
- Large scale evaluation of importance maps in automatic speech recognition
- Information-Theoretic Odometry Learning
- Off-Policy Meta-Reinforcement Learning Based on Feature Embedding Spaces
- Privacy-Constrained Policies via Mutual Information Regularized Policy Gradients
- Measuring Progress in Deep Reinforcement Learning Sample Efficiency
- Attentional Bottleneck: Towards an Interpretable Deep Driving Network
- Instance based Generalization in Reinforcement Learning
- Reinforced Natural Language Interfaces via Entropy Decomposition