Variational Option Discovery Algorithms

Abstract

We explore methods for option discovery based on variational inference and make two algorithmic contributions. First: we highlight a tight connection between variational option discovery methods and variational autoencoders, and introduce Variational Autoencoding Learning of Options by Reinforcement (VALOR), a new method derived from the connection. In VALOR, the policy encodes contexts from a noise distribution into trajectories, and the decoder recovers the contexts from the complete trajectories. Second: we propose a curriculum learning approach where the number of contexts seen by the agent increases whenever the agent's performance is strong enough (as measured by the decoder) on the current set of contexts. We show that this simple trick stabilizes training for VALOR and prior variational option discovery methods, allowing a single agent to learn many more modes of behavior than it could with a fixed context distribution. Finally, we investigate other topics related to variational option discovery, including fundamental limitations of the general approach and the applicability of learned options to downstream tasks.

References in corpus (6)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research
• Eigenoption Discovery through the Deep Successor Representation
• Multi-Level Discovery of Deep Options
• The Eigenoption-Critic Framework

Cited by in corpus (36)

• Exploration by Random Network Distillation
• Agent57: Outperforming the Atari Human Benchmark
• Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain
• A survey on intrinsic motivation in reinforcement learning
• Efficient Exploration via State Marginal Matching
• Dynamics-Aware Unsupervised Discovery of Skills
• Unsupervised Meta-Learning for Reinforcement Learning
• Action and Perception as Divergence Minimization
• Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills
• PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training
• Reset-Free Lifelong Learning with Skill-Space Planning
• Emergent Real-World Robotic Skills via Unsupervised Off-Policy Reinforcement Learning
• Co-GAIL: Learning Diverse Strategies for Human-Robot Collaboration
• Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives
• Preventing Imitation Learning with Adversarial Policy Ensembles
• Learning more skills through optimistic exploration
• Constraint Satisfaction Propagation: Non-stationary Policy Synthesis for Temporal Logic Planning
• Skill Discovery of Coordination in Multi-agent Reinforcement Learning
• IR-VIC: Unsupervised Discovery of Sub-goals for Transfer in RL
• Learning Task Decomposition with Ordered Memory Policy Network
• Adaptable Agent Populations via a Generative Model of Policies
• Affordance as general value function: A computational model
• The Information Geometry of Unsupervised Reinforcement Learning
• Continual Reinforcement Learning with Diversity Exploration and Adversarial Self-Correction
• Intrinsically motivated option learning: a comparative study of recent methods
• Hierarchical Variational Imitation Learning of Control Programs
• Hierarchical Skills for Efficient Exploration
• Reward is enough for convex MDPs
• Discovering Diverse Athletic Jumping Strategies
• Direct then Diffuse: Incremental Unsupervised Skill Discovery for State Covering and Goal Reaching
• PODNet: A Neural Network for Discovery of Plannable Options
• From proprioception to long-horizon planning in novel environments: A hierarchical RL model
• Hard Attention Control By Mutual Information Maximization
• Policy Manifold Search for Improving Diversity-based Neuroevolution
• Unsupervised Skill-Discovery and Skill-Learning in Minecraft
• Neural Embedding for Physical Manipulations