Benchmarking Reinforcement Learning Algorithms on Real-World Robots

Abstract

Through many recent successes in simulation, model-free reinforcement learning has emerged as a promising approach to solving continuous control robotic tasks. The research community is now able to reproduce, analyze and build quickly on these results due to open source implementations of learning algorithms and simulated benchmark tasks. To carry forward these successes to real-world applications, it is crucial to withhold utilizing the unique advantages of simulations that do not transfer to the real world and experiment directly with physical robots. However, reinforcement learning research with physical robots faces substantial resistance due to the lack of benchmark tasks and supporting source code. In this work, we introduce several reinforcement learning tasks with multiple commercially available robots that present varying levels of learning difficulty, setup, and repeatability. On these tasks, we test the learning performance of off-the-shelf implementations of four reinforcement learning algorithms and analyze sensitivity to their hyper-parameters to determine their readiness for applications in various real-world tasks. Our results show that with a careful setup of the task interface and computations, some of these implementations can be readily applicable to physical robots. We find that state-of-the-art learning algorithms are highly sensitive to their hyper-parameters and their relative ordering does not transfer across tasks, indicating the necessity of re-tuning them for each task for best performance. On the other hand, the best hyper-parameter configuration from one task may often result in effective learning on held-out tasks even with different robots, providing a reasonable default. We make the benchmark tasks publicly available to enhance reproducibility in real-world reinforcement learning.

arXiv commentAppears in Proceedings of the Second Conference on Robot Learning (CoRL 2018). Companion video at https://youtu.be/ovDfhvjpQd8 and source code at https://github.com/kindredresearch/SenseAct

Cited by in corpus (13)

• Learning Force Control for Contact-rich Manipulation Tasks with Rigid Position-controlled Robots
• How to pick the domain randomization parameters for sim-to-real transfer of reinforcement learning policies?
• Learning a Low-dimensional Representation of a Safe Region for Safe Reinforcement Learning on Dynamical Systems
• Whole-Body Control of a Mobile Manipulator using End-to-End Reinforcement Learning
• Fast Efficient Hyperparameter Tuning for Policy Gradients
• Sample-Efficient Automated Deep Reinforcement Learning
• Obstacle Avoidance and Navigation Utilizing Reinforcement Learning with Reward Shaping
• SENTINEL: Taming Uncertainty with Ensemble-based Distributional Reinforcement Learning
• Bridging Scene Understanding and Task Execution with Flexible Simulation Environments
• Deep Reinforcement Learning for Tactile Robotics: Learning to Type on a Braille Keyboard
• Model-free Policy Learning with Reward Gradients
• How to Train your Quadrotor: A Framework for Consistently Smooth and Responsive Flight Control via Reinforcement Learning
• GrowSpace: Learning How to Shape Plants