Variational Intrinsic Control

Abstract

In this paper we introduce a new unsupervised reinforcement learning method for discovering the set of intrinsic options available to an agent. This set is learned by maximizing the number of different states an agent can reliably reach, as measured by the mutual information between the set of options and option termination states. To this end, we instantiate two policy gradient based algorithms, one that creates an explicit embedding space of options and one that represents options implicitly. The algorithms also provide an explicit measure of empowerment in a given state that can be used by an empowerment maximizing agent. The algorithm scales well with function approximation and we demonstrate the applicability of the algorithm on a range of tasks.

15 pages, 6 figures

References in corpus (4)

• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• VIME: Variational Information Maximizing Exploration
• Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
• Strategic Attentive Writer for Learning Macro-Actions

Cited by in corpus (86)

• How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned
• Large-Scale Study of Curiosity-Driven Learning
• Agent57: Outperforming the Atari Human Benchmark
• Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain
• Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning
• Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learning
• A survey on intrinsic motivation in reinforcement learning
• Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings
• Dynamics-Aware Unsupervised Discovery of Skills
• RODE: Learning Roles to Decompose Multi-Agent Tasks
• Unsupervised Meta-Learning for Reinforcement Learning
• Routing Networks and the Challenges of Modular and Compositional Computation
• Reward learning from human preferences and demonstrations in Atari
• Automated curricula through setter-solver interactions
• From Machine Learning to Robotics: Challenges and Opportunities for Embodied Intelligence
• A unified strategy for implementing curiosity and empowerment driven reinforcement learning
• Hierarchical Reinforcement Learning By Discovering Intrinsic Options
• Action and Perception as Divergence Minimization
• Penalizing side effects using stepwise relative reachability
• Fast Task Inference with Variational Intrinsic Successor Features
• Exploiting Hierarchy for Learning and Transfer in KL-regularized RL
• BeBold: Exploration Beyond the Boundary of Explored Regions
• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills
• AvE: Assistance via Empowerment
• Learning Goal Embeddings via Self-Play for Hierarchical Reinforcement Learning
• Dynamics-Aware Quality-Diversity for Efficient Learning of Skill Repertoires
• Beyond Fine-Tuning: Transferring Behavior in Reinforcement Learning
• Reset-Free Lifelong Learning with Skill-Space Planning
• MADE: Exploration via Maximizing Deviation from Explored Regions
• Behavior From the Void: Unsupervised Active Pre-Training
• Curiosity-Driven Multi-Criteria Hindsight Experience Replay
• Evaluating Agents without Rewards
• Towards General and Autonomous Learning of Core Skills: A Case Study in Locomotion
• One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL
• Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrifices
• Geometric Entropic Exploration
• Which Mutual-Information Representation Learning Objectives are Sufficient for Control?
• The Variational Bandwidth Bottleneck: Stochastic Evaluation on an Information Budget
• Is Curiosity All You Need? On the Utility of Emergent Behaviours from Curious Exploration
• Temporal Difference Uncertainties as a Signal for Exploration
• Novelty Search in Representational Space for Sample Efficient Exploration
• Co-GAIL: Learning Diverse Strategies for Human-Robot Collaboration
• Diversity-Enriched Option-Critic
• Task-Agnostic Morphology Evolution
• TRAIL: Near-Optimal Imitation Learning with Suboptimal Data
• Learning Efficient Representation for Intrinsic Motivation
• Discovery of Options via Meta-Learned Subgoals
• Learning more skills through optimistic exploration
• Implicit Generative Modeling for Efficient Exploration
• Mutual Information State Intrinsic Control
• Diversity-Driven Extensible Hierarchical Reinforcement Learning
• Robust Multi-Agent Reinforcement Learning with Social Empowerment for Coordination and Communication
• The Journey is the Reward: Unsupervised Learning of Influential Trajectories
• New And Surprising Ways to Be Mean. Adversarial NPCs with Coupled Empowerment Minimisation
• Skill Discovery of Coordination in Multi-agent Reinforcement Learning
• IR-VIC: Unsupervised Discovery of Sub-goals for Transfer in RL
• Curiosity creates Diversity in Policy Search
• Touch-based Curiosity for Sparse-Reward Tasks
• Understanding the Origin of Information-Seeking Exploration in Probabilistic Objectives for Control
• Policy Information Capacity: Information-Theoretic Measure for Task Complexity in Deep Reinforcement Learning
• Continual Reinforcement Learning with Diversity Exploration and Adversarial Self-Correction
• Affordance as general value function: A computational model
• Efficient Empowerment Estimation for Unsupervised Stabilization
• Learning Compositional Neural Programs for Continuous Control
• The Information Geometry of Unsupervised Reinforcement Learning
• Mutual Information-based State-Control for Intrinsically Motivated Reinforcement Learning
• Options Discovery with Budgeted Reinforcement Learning
• Action Redundancy in Reinforcement Learning
• Learning body-affordances to simplify action spaces
• Intrinsically motivated option learning: a comparative study of recent methods
• Towards Robust Bisimulation Metric Learning
• Reward is enough for convex MDPs
• Learning Altruistic Behaviours in Reinforcement Learning without External Rewards
• Direct then Diffuse: Incremental Unsupervised Skill Discovery for State Covering and Goal Reaching
• Bottom-Up Skill Discovery from Unsegmented Demonstrations for Long-Horizon Robot Manipulation
• Discovering a set of policies for the worst case reward
• From proprioception to long-horizon planning in novel environments: A hierarchical RL model
• When should agents explore?
• Hard Attention Control By Mutual Information Maximization
• Learning Task Agnostic Skills with Data-driven Guidance
• Experimental Evidence that Empowerment May Drive Exploration in Sparse-Reward Environments
• Explore and Control with Adversarial Surprise
• Wasserstein Distance Maximizing Intrinsic Control
• Variational Intrinsic Control Revisited
• Understanding the World Through Action