Reinforcement Learning for Robotic Manipulation using Simulated Locomotion Demonstrations
arXiv:1910.07294 · doi:10.1007/s10994-021-06116-1
Abstract
Mastering robotic manipulation skills through reinforcement learning (RL) typically requires the design of shaped reward functions. Recent developments in this area have demonstrated that using sparse rewards, i.e. rewarding the agent only when the task has been successfully completed, can lead to better policies. However, state-action space exploration is more difficult in this case. Recent RL approaches to learning with sparse rewards have leveraged high-quality human demonstrations for the task, but these can be costly, time consuming or even impossible to obtain. In this paper, we propose a novel and effective approach that does not require human demonstrations. We observe that every robotic manipulation task could be seen as involving a locomotion task from the perspective of the object being manipulated, i.e. the object could learn how to reach a target state on its own. In order to exploit this idea, we introduce a framework whereby an object locomotion policy is initially obtained using a realistic physics simulator. This policy is then used to generate auxiliary rewards, called simulated locomotion demonstration rewards (SLDRs), which enable us to learn the robot manipulation policy. The proposed approach has been evaluated on 13 tasks of increasing complexity, and can achieve higher success rate and faster learning rates compared to alternative algorithms. SLDRs are especially beneficial for tasks like multi-object stacking and non-rigid object manipulation.
arXiv commentTo appear in ECML PKDD 2022
References in corpus (24)
- Continuous control with deep reinforcement learning
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- End to End Learning for Self-Driving Cars
- Trust Region Policy Optimization
- Asynchronous Methods for Deep Reinforcement Learning
- Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
- VIME: Variational Information Maximizing Exploration
- Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
- Large-Scale Study of Curiosity-Driven Learning
- Visual Reinforcement Learning with Imagined Goals
- Deep Q-learning from Demonstrations
- Learning Dexterous In-Hand Manipulation
- Learning by Playing - Solving Sparse Reward Tasks from Scratch
- Exploration by Random Network Distillation
- Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research
- Episodic Curiosity through Reachability
- Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
- Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids
- Automatic Goal Generation for Reinforcement Learning Agents
- SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning
- Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play
- Data-efficient Deep Reinforcement Learning for Dexterous Manipulation
- Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations
- Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Cost