Scaling data-driven robotics with reward sketching and batch reinforcement learning

Abstract

We present a framework for data-driven robotics that makes use of a large dataset of recorded robot experience and scales to several tasks using learned reward functions. We show how to apply this framework to accomplish three different object manipulation tasks on a real robot platform. Given demonstrations of a task together with task-agnostic recorded experience, we use a special form of human annotation as supervision to learn a reward function, which enables us to deal with real-world tasks where the reward signal cannot be acquired directly. Learned rewards are used in combination with a large dataset of experience from different tasks to learn a robot policy offline using batch RL. We show that using our approach it is possible to train agents to perform a variety of challenging manipulation tasks including stacking rigid objects and handling cloth.

Project website: https://sites.google.com/view/data-driven-robotics/

References in corpus (18)

• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Dota 2 with Large Scale Deep Reinforcement Learning
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• Distributed Distributional Deterministic Policy Gradients
• Learning Dexterous In-Hand Manipulation
• Challenges of Real-World Reinforcement Learning
• Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World
• Learning human behaviors from motion capture by adversarial imitation
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Task
• Sim-to-Real Reinforcement Learning for Deformable Object Manipulation
• Observe and Look Further: Achieving Consistent Performance on Atari
• Acme: A Research Framework for Distributed Reinforcement Learning
• Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitation
• Self-Supervised Sim-to-Real Adaptation for Visual Robotic Manipulation

Cited by in corpus (23)

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Critic Regularized Regression
• What Matters in Learning from Offline Human Demonstrations for Robot Manipulation
• An empirical investigation of the challenges of real-world reinforcement learning
• Imitating Interactive Intelligence
• Learning to summarize from human feedback
• MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale
• ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations
• Learning Arbitrary-Goal Fabric Folding with One Hour of Real Robot Experience
• GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement
• The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixels
• Multi-task Batch Reinforcement Learning with Metric Learning
• Human-in-the-Loop Imitation Learning using Remote Teleoperation
• Quantifying Differences in Reward Functions
• Positive-Unlabeled Reward Learning
• Offline Meta-Reinforcement Learning with Online Self-Supervision
• Learning Dense Rewards for Contact-Rich Manipulation Tasks
• Fighting Failures with FIRE: Failure Identification to Reduce Expert Burden in Intervention-Based Learning
• S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning
• Robust Multi-Modal Policies for Industrial Assembly via Reinforcement Learning and Demonstrations: A Large-Scale Study
• Offline Preference-Based Apprenticeship Learning
• "What, not how": Solving an under-actuated insertion task from scratch