REPLAB: A Reproducible Low-Cost Arm Benchmark Platform for Robotic Learning

Abstract

Standardized evaluation measures have aided in the progress of machine learning approaches in disciplines such as computer vision and machine translation. In this paper, we make the case that robotic learning would also benefit from benchmarking, and present the "REPLAB" platform for benchmarking vision-based manipulation tasks. REPLAB is a reproducible and self-contained hardware stack (robot arm, camera, and workspace) that costs about 2000 USD, occupies a cuboid of size 70x40x60 cm, and permits full assembly within a few hours. Through this low-cost, compact design, REPLAB aims to drive wide participation by lowering the barrier to entry into robotics and to enable easy scaling to many robots. We envision REPLAB as a framework for reproducible research across manipulation tasks, and as a step in this direction, we define a template for a grasping benchmark consisting of a task definition, evaluation protocol, performance measures, and a dataset of 92k grasp attempts. We implement, evaluate, and analyze several previously proposed grasping approaches to establish baselines for this benchmark. Finally, we also implement and evaluate a deep reinforcement learning approach for 3D reaching tasks on our REPLAB platform. Project page with assembly instructions, code, and videos: https://goo.gl/5F9dP4.

Extended version of paper accepted to ICRA 2019

References in corpus (9)

• Addressing Function Approximation Error in Actor-Critic Methods
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
• Deep Reinforcement Learning that Matters
• Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics
• Learning Dexterous In-Hand Manipulation
• Robot Learning in Homes: Improving Generalization and Reducing Dataset Bias
• Robotic Pick-and-Place of Novel Objects in Clutter with Multi-Affordance Grasping and Cross-Domain Image Matching
• Using Geometry to Detect Grasps in 3D Point Clouds

Cited by in corpus (11)

• PyRobot: An Open-source Robotics Framework for Research and Benchmarking
• TriFinger: An Open-Source Robot for Learning Dexterity
• Rigid-Soft Interactive Learning for Robust Grasping
• ROBEL: Robotics Benchmarks for Learning with Low-Cost Robots
• OffWorld Gym: open-access physical robotics environment for real-world reinforcement learning benchmark and research
• rl_reach: Reproducible Reinforcement Learning Experiments for Robotic Reaching Tasks
• Interferobot: aligning an optical interferometer by a reinforcement learning agent
• Measuring Progress in Deep Reinforcement Learning Sample Efficiency
• DeepClaw: A Robotic Hardware Benchmarking Platform for Learning Object Manipulation
• Simultaneous Navigation and Construction Benchmarking Environments
• Morphology-Agnostic Visual Robotic Control