Emergent Real-World Robotic Skills via Unsupervised Off-Policy Reinforcement Learning

Abstract

Reinforcement learning provides a general framework for learning robotic skills while minimizing engineering effort. However, most reinforcement learning algorithms assume that a well-designed reward function is provided, and learn a single behavior for that single reward function. Such reward functions can be difficult to design in practice. Can we instead develop efficient reinforcement learning methods that acquire diverse skills without any reward function, and then repurpose these skills for downstream tasks? In this paper, we demonstrate that a recently proposed unsupervised skill discovery algorithm can be extended into an efficient off-policy method, making it suitable for performing unsupervised reinforcement learning in the real world. Firstly, we show that our proposed algorithm provides substantial improvement in learning efficiency, making reward-free real-world training feasible. Secondly, we move beyond the simulation environments and evaluate the algorithm on real physical hardware. On quadrupeds, we observe that locomotion skills with diverse gaits and different orientations emerge without any rewards or demonstrations. We also demonstrate that the learned skills can be composed using model predictive control for goal-oriented navigation, without any additional training.

References in corpus (14)

• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Soft Actor-Critic Algorithms and Applications
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Deep reinforcement learning from human preferences
• Stochastic Neural Networks for Hierarchical Reinforcement Learning
• Efficient Exploration via State Marginal Matching
• Doubly Robust Off-policy Value Evaluation for Reinforcement Learning
• Variational Option Discovery Algorithms
• Dynamics-Aware Unsupervised Discovery of Skills
• Deep Dynamics Models for Learning Dexterous Manipulation
• CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learning
• ROBEL: Robotics Benchmarks for Learning with Low-Cost Robots

Cited by in corpus (7)

• Reset-Free Lifelong Learning with Skill-Space Planning
• Multi-task Batch Reinforcement Learning with Metric Learning
• Towards General and Autonomous Learning of Core Skills: A Case Study in Locomotion
• Is Curiosity All You Need? On the Utility of Emergent Behaviours from Curious Exploration
• Wish you were here: Hindsight Goal Selection for long-horizon dexterous manipulation
• Autonomous Reinforcement Learning via Subgoal Curricula
• Adaptation of Quadruped Robot Locomotion with Meta-Learning