Towards Practical Multi-Object Manipulation using Relational Reinforcement Learning

Abstract

Learning robotic manipulation tasks using reinforcement learning with sparse rewards is currently impractical due to the outrageous data requirements. Many practical tasks require manipulation of multiple objects, and the complexity of such tasks increases with the number of objects. Learning from a curriculum of increasingly complex tasks appears to be a natural solution, but unfortunately, does not work for many scenarios. We hypothesize that the inability of the state-of-the-art algorithms to effectively utilize a task curriculum stems from the absence of inductive biases for transferring knowledge from simpler to complex tasks. We show that graph-based relational architectures overcome this limitation and enable learning of complex tasks when provided with a simple curriculum of tasks with increasing numbers of objects. We demonstrate the utility of our framework on a simulated block stacking task. Starting from scratch, our agent learns to stack six blocks into a tower. Despite using step-wise sparse rewards, our method is orders of magnitude more data-efficient and outperforms the existing state-of-the-art method that utilizes human demonstrations. Furthermore, the learned policy exhibits zero-shot generalization, successfully stacking blocks into taller towers and previously unseen configurations such as pyramids, without any further training.

10 pages, 4 figures and 1 table in main article, 3 figures and 3 tables in appendix. Supplementary website and videos at https://richardrl.github.io/relational-rl/

References in corpus (19)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Continuous control with deep reinforcement learning
• Semi-Supervised Classification with Graph Convolutional Networks
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• Interaction Networks for Learning about Objects, Relations and Physics
• On First-Order Meta-Learning Algorithms
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Learning to Execute
• Learning Dexterous In-Hand Manipulation
• A Study on Overfitting in Deep Reinforcement Learning
• Quantifying Generalization in Reinforcement Learning
• Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research
• Automatic Goal Generation for Reinforcement Learning Agents
• Reverse Curriculum Generation for Reinforcement Learning
• Data-efficient Deep Reinforcement Learning for Dexterous Manipulation
• Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation
• Divide-and-Conquer Reinforcement Learning
• Modulated Policy Hierarchies

Cited by in corpus (10)

• Asymmetric self-play for automatic goal discovery in robotic manipulation
• Grounding Language to Autonomously-Acquired Skills via Goal Generation
• Counterfactual Data Augmentation using Locally Factored Dynamics
• Learning to Optimize Industry-Scale Dynamic Pickup and Delivery Problems
• Machine Learning for Robotic Manipulation
• Efficient and Interpretable Robot Manipulation with Graph Neural Networks
• Learning Compositional Neural Programs for Continuous Control
• Policy learning in SE(3) action spaces
• Flexible and Efficient Long-Range Planning Through Curious Exploration
• Learning to Design and Construct Bridge without Blueprint