The Ingredients of Real-World Robotic Reinforcement Learning

Abstract

The success of reinforcement learning for real world robotics has been, in many cases limited to instrumented laboratory scenarios, often requiring arduous human effort and oversight to enable continuous learning. In this work, we discuss the elements that are needed for a robotic learning system that can continually and autonomously improve with data collected in the real world. We propose a particular instantiation of such a system, using dexterous manipulation as our case study. Subsequently, we investigate a number of challenges that come up when learning without instrumentation. In such settings, learning must be feasible without manually designed resets, using only on-board perception, and without hand-engineered reward functions. We propose simple and scalable solutions to these challenges, and then demonstrate the efficacy of our proposed system on a set of dexterous robotic manipulation tasks, providing an in-depth analysis of the challenges associated with this learning paradigm. We demonstrate that our complete system can learn without any human intervention, acquiring a variety of vision-based skills with a real-world three-fingered hand. Results and videos can be found at https://sites.google.com/view/realworld-rl/

arXiv commentFirst three authors contributed equally. Accepted as a spotlight presentation at ICLR 2020

References in corpus (10)

• Continuous control with deep reinforcement learning
• Trust Region Policy Optimization
• Soft Actor-Critic Algorithms and Applications
• Learning deep representations by mutual information estimation and maximization
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• Robust Adversarial Reinforcement Learning
• Exploration by Random Network Distillation
• Combining Model-Based and Model-Free Updates for Trajectory-Centric Reinforcement Learning
• End-to-End Robotic Reinforcement Learning without Reward Engineering

Cited by in corpus (4)

• TriFinger: An Open-Source Robot for Learning Dexterity
• GST: Group-Sparse Training for Accelerating Deep Reinforcement Learning
• Multitask Adaptation by Retrospective Exploration with Learned World Models
• Learning of feature points without additional supervision improves reinforcement learning from images