Balance Between Efficient and Effective Learning: Dense2Sparse Reward Shaping for Robot Manipulation with Environment Uncertainty
arXiv:2003.02740
Abstract
Efficient and effective learning is one of the ultimate goals of the deep reinforcement learning (DRL), although the compromise has been made in most of the time, especially for the application of robot manipulations. Learning is always expensive for robot manipulation tasks and the learning effectiveness could be affected by the system uncertainty. In order to solve above challenges, in this study, we proposed a simple but powerful reward shaping method, namely Dense2Sparse. It combines the advantage of fast convergence of dense reward and the noise isolation of the sparse reward, to achieve a balance between learning efficiency and effectiveness, which makes it suitable for robot manipulation tasks. We evaluated our Dense2Sparse method with a series of ablation experiments using the state representation model with system uncertainty. The experiment results show that the Dense2Sparse method obtained higher expected reward compared with the ones using standalone dense reward or sparse reward, and it also has a superior tolerance of system uncertainty.
References in corpus (15)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Continuous control with deep reinforcement learning
- Playing Atari with Deep Reinforcement Learning
- Addressing Function Approximation Error in Actor-Critic Methods
- QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
- Emergence of Locomotion Behaviours in Rich Environments
- Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
- Distributed Distributional Deterministic Policy Gradients
- Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Task
- Sim-to-Real: Learning Agile Locomotion For Quadruped Robots
- Preparing for the Unknown: Learning a Universal Policy with Online System Identification
- Reward Shaping via Meta-Learning
- Reinforcement Learning for Pivoting Task
- Reinforcement Learning with Perturbed Rewards
- Motion Perception in Reinforcement Learning with Dynamic Objects