Discovering Generalizable Skills via Automated Generation of Diverse Tasks

Abstract

The learning efficiency and generalization ability of an intelligent agent can be greatly improved by utilizing a useful set of skills. However, the design of robot skills can often be intractable in real-world applications due to the prohibitive amount of effort and expertise that it requires. In this work, we introduce Skill Learning In Diversified Environments (SLIDE), a method to discover generalizable skills via automated generation of a diverse set of tasks. As opposed to prior work on unsupervised discovery of skills which incentivizes the skills to produce different outcomes in the same environment, our method pairs each skill with a unique task produced by a trainable task generator. To encourage generalizable skills to emerge, our method trains each skill to specialize in the paired task and maximizes the diversity of the generated tasks. A task discriminator defined on the robot behaviors in the generated tasks is jointly trained to estimate the evidence lower bound of the diversity objective. The learned skills can then be composed in a hierarchical reinforcement learning algorithm to solve unseen target tasks. We demonstrate that the proposed method can effectively learn a variety of robot skills in two tabletop manipulation domains. Our results suggest that the learned skills can effectively improve the robot's performance in various unseen target tasks compared to existing reinforcement learning and skill learning methods.

arXiv commentRSS 2021

References in corpus (21)

• Adam: A Method for Stochastic Optimization
• Emergence of Locomotion Behaviours in Rich Environments
• Solving Rubik's Cube with a Robot Hand
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• AI2-THOR: An Interactive 3D Environment for Visual AI
• Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics
• Modular Multitask Reinforcement Learning with Policy Sketches
• Stochastic Neural Networks for Hierarchical Reinforcement Learning
• Leveraging Procedural Generation to Benchmark Reinforcement Learning
• Procedural Content Generation through Quality Diversity
• Meta Learning Shared Hierarchies
• Mechanical Search: Multi-Step Retrieval of a Target Object Occluded by Clutter
• Rearrangement: A Challenge for Embodied AI
• Unsupervised Meta-Learning for Reinforcement Learning
• Disentangling the independently controllable factors of variation by interacting with the world
• Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions
• Active Domain Randomization
• Learning Self-Imitating Diverse Policies
• Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environments
• An Optimal Online Method of Selecting Source Policies for Reinforcement Learning
• Hierarchical Planning for Long-Horizon Manipulation with Geometric and Symbolic Scene Graphs