Dynamics-Aware Unsupervised Discovery of Skills

Abstract

Conventionally, model-based reinforcement learning (MBRL) aims to learn a global model for the dynamics of the environment. A good model can potentially enable planning algorithms to generate a large variety of behaviors and solve diverse tasks. However, learning an accurate model for complex dynamical systems is difficult, and even then, the model might not generalize well outside the distribution of states on which it was trained. In this work, we combine model-based learning with model-free learning of primitives that make model-based planning easy. To that end, we aim to answer the question: how can we discover skills whose outcomes are easy to predict? We propose an unsupervised learning algorithm, Dynamics-Aware Discovery of Skills (DADS), which simultaneously discovers predictable behaviors and learns their dynamics. Our method can leverage continuous skill spaces, theoretically, allowing us to learn infinitely many behaviors even for high-dimensional state-spaces. We demonstrate that zero-shot planning in the learned latent space significantly outperforms standard MBRL and model-free goal-conditioned RL, can handle sparse-reward tasks, and substantially improves over prior hierarchical RL methods for unsupervised skill discovery.

References in corpus (25)

• Adam: A Method for Stochastic Optimization
• Playing Atari with Deep Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• Soft Actor-Critic Algorithms and Applications
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Emergence of Locomotion Behaviours in Rich Environments
• Recurrent World Models Facilitate Policy Evolution
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Hindsight Experience Replay
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
• Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
• Model-Ensemble Trust-Region Policy Optimization
• Data-Efficient Hierarchical Reinforcement Learning
• Stochastic Neural Networks for Hierarchical Reinforcement Learning
• Variational Intrinsic Control
• Data-Efficient Reinforcement Learning with Probabilistic Model Predictive Control
• Multivariate Information Bottleneck
• Learning and Transfer of Modulated Locomotor Controllers
• Combining Model-Based and Model-Free Updates for Trajectory-Centric Reinforcement Learning
• Variational Option Discovery Algorithms
• EX2: Exploration with Exemplar Models for Deep Reinforcement Learning
• Estimating mutual information and multi--information in large networks
• TherML: Thermodynamics of Machine Learning

Cited by in corpus (38)

• A survey on intrinsic motivation in reinforcement learning
• Planning to Explore via Self-Supervised World Models
• Unsupervised Meta-Learning for Reinforcement Learning
• Adaptive Prior Selection for Repertoire-based Online Adaptation in Robotics
• Hierarchical Reinforcement Learning By Discovering Intrinsic Options
• Action and Perception as Divergence Minimization
• Phase Transitions for the Information Bottleneck in Representation Learning
• OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning
• Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills
• Dynamics-Aware Quality-Diversity for Efficient Learning of Skill Repertoires
• Model-Predictive Control via Cross-Entropy and Gradient-Based Optimization
• Reset-Free Lifelong Learning with Skill-Space Planning
• Emergent Real-World Robotic Skills via Unsupervised Off-Policy Reinforcement Learning
• Evaluating Agents without Rewards
• One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL
• Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks
• Is Curiosity All You Need? On the Utility of Emergent Behaviours from Curious Exploration
• SOAC: The Soft Option Actor-Critic Architecture
• TRAIL: Near-Optimal Imitation Learning with Suboptimal Data
• Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives
• Task-Agnostic Morphology Evolution
• Intelligence, physics and information -- the tradeoff between accuracy and simplicity in machine learning
• Simple Sensor Intentions for Exploration
• Hierarchies of Planning and Reinforcement Learning for Robot Navigation
• Disentangling causal effects for hierarchical reinforcement learning
• Novel Policy Seeking with Constrained Optimization
• Pretrained Encoders are All You Need
• Discovering Diverse Athletic Jumping Strategies
• PAC Bounds for Imitation and Model-based Batch Learning of Contextual Markov Decision Processes
• No Representation without Transformation
• Distilling a Hierarchical Policy for Planning and Control via Representation and Reinforcement Learning
• Policy Manifold Search for Improving Diversity-based Neuroevolution
• Video2Skill: Adapting Events in Demonstration Videos to Skills in an Environment using Cyclic MDP Homomorphisms
• Hard Attention Control By Mutual Information Maximization
• Experimental Evidence that Empowerment May Drive Exploration in Sparse-Reward Environments
• Discriminator Augmented Model-Based Reinforcement Learning
• From proprioception to long-horizon planning in novel environments: A hierarchical RL model
• Understanding the World Through Action