Asymmetric self-play for automatic goal discovery in robotic manipulation

Abstract

We train a single, goal-conditioned policy that can solve many robotic manipulation tasks, including tasks with previously unseen goals and objects. We rely on asymmetric self-play for goal discovery, where two agents, Alice and Bob, play a game. Alice is asked to propose challenging goals and Bob aims to solve them. We show that this method can discover highly diverse and complex goals without any human priors. Bob can be trained with only sparse rewards, because the interaction between Alice and Bob results in a natural curriculum and Bob can learn from Alice's trajectory when relabeled as a goal-conditioned demonstration. Finally, our method scales, resulting in a single policy that can generalize to many unseen tasks such as setting a table, stacking blocks, and solving simple puzzles. Videos of a learned policy is available at https://robotics-self-play.github.io.

arXiv commentVideos are shown at https://robotics-self-play.github.io

References in corpus (21)

• Adam: A Method for Stochastic Optimization
• Learning agile and dynamic motor skills for legged robots
• Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
• Dota 2 with Large Scale Deep Reinforcement Learning
• Solving Rubik's Cube with a Robot Hand
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robots
• Learning to reinforcement learn
• FeUdal Networks for Hierarchical Reinforcement Learning
• Data-Efficient Hierarchical Reinforcement Learning
• Go-Explore: a New Approach for Hard-Exploration Problems
• Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions
• Data-efficient Deep Reinforcement Learning for Dexterous Manipulation
• Sim-to-Real Robot Learning from Pixels with Progressive Nets
• Learning Montezuma's Revenge from a Single Demonstration
• Learning Multi-Level Hierarchies with Hindsight
• Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions
• Automatic Curriculum Learning through Value Disagreement
• Towards Practical Multi-Object Manipulation using Relational Reinforcement Learning
• Learning Goal Embeddings via Self-Play for Hierarchical Reinforcement Learning

Cited by in corpus (10)

• Exploration in Deep Reinforcement Learning: A Survey
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• A social path to human-like artificial intelligence
• ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations
• Multi-task curriculum learning in a complex, visual, hard-exploration domain: Minecraft
• Manipulator-Independent Representations for Visual Imitation
• Stable Object Reorientation using Contact Plane Registration
• Replay-Guided Adversarial Environment Design
• Explore and Control with Adversarial Surprise
• Room Clearance with Feudal Hierarchical Reinforcement Learning