VIME: Variational Information Maximizing Exploration

Abstract

Scalable and effective exploration remains a key challenge in reinforcement learning (RL). While there are methods with optimality guarantees in the setting of discrete state and action spaces, these methods cannot be applied in high-dimensional deep RL scenarios. As such, most contemporary RL relies on simple heuristics such as epsilon-greedy exploration or adding Gaussian noise to the controls. This paper introduces Variational Information Maximizing Exploration (VIME), an exploration strategy based on maximization of information gain about the agent's belief of environment dynamics. We propose a practical implementation, using variational inference in Bayesian neural networks which efficiently handles continuous state and action spaces. VIME modifies the MDP reward function, and can be applied with several different underlying RL algorithms. We demonstrate that VIME achieves significantly better performance compared to heuristic exploration methods across a variety of continuous control tasks and algorithms, including tasks with very sparse rewards.

arXiv commentPublished in Advances in Neural Information Processing Systems 29 (NIPS), pages 1109-1117

References in corpus (9)

• Weight Uncertainty in Neural Networks
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Deep Exploration via Bootstrapped DQN
• Action-Conditional Video Prediction using Deep Networks in Atari Games
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
• Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
• Generalization and Exploration via Randomized Value Functions
• Information Theoretically Aided Reinforcement Learning for Embodied Agents
• Simple Algorithmic Principles of Discovery, Subjective Beauty, Selective Attention, Curiosity & Creativity

Cited by in corpus (149)

• A Brief Survey of Deep Reinforcement Learning
• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• An Introduction to Variational Autoencoders
• DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
• Noisy Networks for Exploration
• Deep Reinforcement Learning: An Overview
• Exploration in Deep Reinforcement Learning: A Survey
• Emergent Tool Use From Multi-Agent Autocurricula
• Large-Scale Study of Curiosity-Driven Learning
• Count-Based Exploration with Neural Density Models
• Hindsight Experience Replay
• Unifying Count-Based Exploration and Intrinsic Motivation
• Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control
• Stochastic Neural Networks for Hierarchical Reinforcement Learning
• Episodic Curiosity through Reachability
• Variational Intrinsic Control
• Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain
• Meta-Reinforcement Learning of Structured Exploration Strategies
• Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning
• Bayesian Model-Agnostic Meta-Learning
• Automatic Goal Generation for Reinforcement Learning Agents
• Bayesian Recurrent Neural Networks
• Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents
• Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning
• Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learning
• A survey on intrinsic motivation in reinforcement learning
• Loss is its own Reward: Self-Supervision for Reinforcement Learning
• Efficient Exploration via State Marginal Matching
• VIBNN: Hardware Acceleration of Bayesian Neural Networks
• Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings
• GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithms
• Stein Variational Policy Gradient
• Some Considerations on Learning to Explore via Meta-Reinforcement Learning
• Curiosity Driven Exploration of Learned Disentangled Goal Spaces
• Discrete Sequential Prediction of Continuous Actions for Deep RL
• Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control
• Reinforcement Learning through Active Inference
• Contingency-Aware Exploration in Reinforcement Learning
• Unsupervised Learning of Goal Spaces for Intrinsically Motivated Goal Exploration
• Rule-Based Reinforcement Learning for Efficient Robot Navigation with Space Reduction
• Unsupervised Meta-Learning for Reinforcement Learning
• An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey
• SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning
• The Free Energy Principle for Perception and Action: A Deep Learning Perspective
• Learning Gentle Object Manipulation with Curiosity-Driven Deep Reinforcement Learning
• InfoBot: Transfer and Exploration via the Information Bottleneck
• Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learning
• Exploiting Oxide Based Resistive RAM Variability for Bayesian Neural Network Hardware Design
• Curiosity-Driven Experience Prioritization via Density Estimation
• Automated Curriculum Learning for Neural Networks
• Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation
• Reinforcement Learning for Robotic Manipulation using Simulated Locomotion Demonstrations
• Coordinated Exploration via Intrinsic Rewards for Multi-Agent Reinforcement Learning
• Learning Multimodal Transition Dynamics for Model-Based Reinforcement Learning
• Trust-PCL: An Off-Policy Trust Region Method for Continuous Control
• Noisy Natural Gradient as Variational Inference
• A Survey of Embodied AI: From Simulators to Research Tasks
• Is Pessimism Provably Efficient for Offline RL?
• Perfect density models cannot guarantee anomaly detection
• Learning to Explore with Meta-Policy Gradient
• Action and Perception as Divergence Minimization
• Learning Self-Imitating Diverse Policies
• Variational Dynamic for Self-Supervised Exploration in Deep Reinforcement Learning
• Deep Intrinsically Motivated Continuous Actor-Critic for Efficient Robotic Visuomotor Skill Learning
• State Entropy Maximization with Random Encoders for Efficient Exploration
• Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learning
• Reward Design in Cooperative Multi-agent Reinforcement Learning for Packet Routing
• Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills
• Efficient exploration with Double Uncertain Value Networks
• Randomized Value Functions via Multiplicative Normalizing Flows
• Curiosity-driven reinforcement learning with homeostatic regulation
• Ecological Reinforcement Learning
• SMiRL: Surprise Minimizing Reinforcement Learning in Unstable Environments
• Entropy Regularization for Mean Field Games with Learning
• Behavior From the Void: Unsupervised Active Pre-Training
• Dual Control of Exploration and Exploitation for Auto-Optimisation Control with Active Learning
• Exploration and preference satisfaction trade-off in reward-free learning
• Uncertainty in Multitask Transfer Learning
• MetaCURE: Meta Reinforcement Learning with Empowerment-Driven Exploration
• Towards Information-Seeking Agents
• Causal Influence Detection for Improving Efficiency in Reinforcement Learning
• Diversity Actor-Critic: Sample-Aware Entropy Regularization for Sample-Efficient Exploration
• Adapting Behaviour via Intrinsic Reward: A Survey and Empirical Study
• Flow Models for Arbitrary Conditional Likelihoods
• Learning-Driven Exploration for Reinforcement Learning
• Long-Term Visitation Value for Deep Exploration in Sparse Reward Reinforcement Learning
• Variational Deep Q Network
• Multimodal Safety-Critical Scenarios Generation for Decision-Making Algorithms Evaluation
• Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrifices
• OffWorld Gym: open-access physical robotics environment for real-world reinforcement learning benchmark and research
• Hindsight Trust Region Policy Optimization
• Dynamic Multi-Level Multi-Task Learning for Sentence Simplification
• Novelty Search in Representational Space for Sample Efficient Exploration
• Curiosity-driven Reinforcement Learning for Diverse Visual Paragraph Generation
• Is Curiosity All You Need? On the Utility of Emergent Behaviours from Curious Exploration
• Stochastic Neural Network with Kronecker Flow
• Instabilities of Offline RL with Pre-Trained Neural Representation
• Locally Persistent Exploration in Continuous Control Tasks with Sparse Rewards
• Epistemic Risk-Sensitive Reinforcement Learning
• Hashing over Predicted Future Frames for Informed Exploration of Deep Reinforcement Learning
• Non-local Policy Optimization via Diversity-regularized Collaborative Exploration
• Flow-based Intrinsic Curiosity Module
• Deep Learning in Robotics: A Review of Recent Research
• A Max-Min Entropy Framework for Reinforcement Learning
• Learning with Stochastic Guidance for Navigation
• Implicit Generative Modeling for Efficient Exploration
• Stochastic Particle-Optimization Sampling and the Non-Asymptotic Convergence Theory
• Model Based Planning with Energy Based Models
• Tutorial and Survey on Probabilistic Graphical Model and Variational Inference in Deep Reinforcement Learning
• Adaptive Procedural Task Generation for Hard-Exploration Problems
• Sensorimotor Visual Perception on Embodied System Using Free Energy Principle
• Dynamical Distance Learning for Semi-Supervised and Unsupervised Skill Discovery
• Micro-Objective Learning : Accelerating Deep Reinforcement Learning through the Discovery of Continuous Subgoals
• Density-based Curriculum for Multi-goal Reinforcement Learning with Sparse Rewards
• Learning Diverse Policies with Soft Self-Generated Guidance
• Understanding the Origin of Information-Seeking Exploration in Probabilistic Objectives for Control
• CLOUD: Contrastive Learning of Unsupervised Dynamics
• Online Safety Assurance for Deep Reinforcement Learning
• An Empowerment-based Solution to Robotic Manipulation Tasks with Sparse Rewards
• Efficient Empowerment Estimation for Unsupervised Stabilization
• Exploration by Maximizing Rényi Entropy for Reward-Free RL Framework
• Novel Policy Seeking with Constrained Optimization
• Mutual Information-based State-Control for Intrinsically Motivated Reinforcement Learning
• Policy Optimization with Model-based Explorations
• Switching Isotropic and Directional Exploration with Parameter Space Noise in Deep Reinforcement Learning
• Self-Paced Deep Reinforcement Learning
• Off-policy Reinforcement Learning with Optimistic Exploration and Distribution Correction
• Discovering Diverse Athletic Jumping Strategies
• GAN-based Intrinsic Exploration For Sample Efficient Reinforcement Learning
• Don't Do What Doesn't Matter: Intrinsic Motivation with Action Usefulness
• Learning to Plan Optimistically: Uncertainty-Guided Deep Exploration via Latent Model Ensembles
• Towards Combining On-Off-Policy Methods for Real-World Applications
• InfoRL: Interpretable Reinforcement Learning using Information Maximization
• Policy Gradient Methods Find the Nash Equilibrium in N-player General-sum Linear-quadratic Games
• ISL: A novel approach for deep exploration
• Leveraging Sequentiality in Reinforcement Learning from a Single Demonstration
• From proprioception to long-horizon planning in novel environments: A hierarchical RL model
• Transfer Learning by Modeling a Distribution over Policies
• Being curious about the answers to questions: novelty search with learned attention
• Shared Learning : Enhancing Reinforcement in $Q$-Ensembles
• Region Growing Curriculum Generation for Reinforcement Learning
• Optimising Stochastic Routing for Taxi Fleets with Model Enhanced Reinforcement Learning
• Decoupled Exploration and Exploitation Policies for Sample-Efficient Reinforcement Learning
• Delayed Rewards Calibration via Reward Empirical Sufficiency
• A Probabilistic Interpretation of Self-Paced Learning with Applications to Reinforcement Learning
• Learning to Shape Rewards using a Game of Two Partners
• Explore and Control with Adversarial Surprise
• When should agents explore?
• Direct then Diffuse: Incremental Unsupervised Skill Discovery for State Covering and Goal Reaching