End-to-End Robotic Reinforcement Learning without Reward Engineering

Abstract

The combination of deep neural network models and reinforcement learning algorithms can make it possible to learn policies for robotic behaviors that directly read in raw sensory inputs, such as camera images, effectively subsuming both estimation and control into one model. However, real-world applications of reinforcement learning must specify the goal of the task by means of a manually programmed reward function, which in practice requires either designing the very same perception pipeline that end-to-end reinforcement learning promises to avoid, or else instrumenting the environment with additional sensors to determine if the task has been performed successfully. In this paper, we propose an approach for removing the need for manual engineering of reward specifications by enabling a robot to learn from a modest number of examples of successful outcomes, followed by actively solicited queries, where the robot shows the user a state and asks for a label to determine whether that state represents successful completion of the task. While requesting labels for every single state would amount to asking the user to manually provide the reward signal, our method requires labels for only a tiny fraction of the states seen during training, making it an efficient and practical approach for learning skills without manually engineered rewards. We evaluate our method on real-world robotic manipulation tasks where the observations consist of images viewed by the robot's camera. In our experiments, our method effectively learns to arrange objects, place books, and drape cloth, directly from images and without any manually specified reward functions, and with only 1-4 hours of interaction with the real world.

Accepted to RSS 2019. 14 pages and 13 figures including references and appendix. Website: https://sites.google.com/view/reward-learning-rl/home

Cited by in corpus (19)

• On the Opportunities and Risks of Foundation Models
• How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned
• COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning
• The Ingredients of Real-World Robotic Reinforcement Learning
• Reward-Conditioned Policies
• PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training
• Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation
• Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks
• To Follow or not to Follow: Selective Imitation Learning from Observations
• Semi-supervised reward learning for offline reinforcement learning
• Lifelong Robotic Reinforcement Learning by Retaining Experiences
• Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classification
• Human-in-the-Loop Methods for Data-Driven and Reinforcement Learning Systems
• Unsupervised Domain Adaptation with Dynamics-Aware Rewards in Reinforcement Learning
• LS3: Latent Space Safe Sets for Long-Horizon Visuomotor Control of Sparse Reward Iterative Tasks
• Perceptual Values from Observation
• Robo-PlaNet: Learning to Poke in a Day
• Rearranging the Environment to Maximize Energy with a Robotic Circuit Drawing
• Learning of feature points without additional supervision improves reinforcement learning from images