Improving Robot Dual-System Motor Learning with Intrinsically Motivated Meta-Control and Latent-Space Experience Imagination

Abstract

Combining model-based and model-free learning systems has been shown to improve the sample efficiency of learning to perform complex robotic tasks. However, dual-system approaches fail to consider the reliability of the learned model when it is applied to make multiple-step predictions, resulting in a compounding of prediction errors and performance degradation. In this paper, we present a novel dual-system motor learning approach where a meta-controller arbitrates online between model-based and model-free decisions based on an estimate of the local reliability of the learned model. The reliability estimate is used in computing an intrinsic feedback signal, encouraging actions that lead to data that improves the model. Our approach also integrates arbitration with imagination where a learned latent-space model generates imagined experiences, based on its local reliability, to be used as additional training data. We evaluate our approach against baseline and state-of-the-art methods on learning vision-based robotic grasping in simulation and real world. The results show that our approach outperforms the compared methods and learns near-optimal grasping policies in dense- and sparse-reward environments.

References in corpus (13)

• Continuous control with deep reinforcement learning
• End-to-End Training of Deep Visuomotor Policies
• Hindsight Experience Replay
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
• Imagination-Augmented Agents for Deep Reinforcement Learning
• Differentiable MPC for End-to-end Planning and Control
• Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
• Provably Efficient Maximum Entropy Exploration
• EX2: Exploration with Exemplar Models for Deep Reinforcement Learning
• Unsupervised Learning of Goal Spaces for Intrinsically Motivated Goal Exploration
• Learning Dynamics Model in Reinforcement Learning by Incorporating the Long Term Future
• Efficient Intrinsically Motivated Robotic Grasping with Learning-Adaptive Imagination in Latent Space

Cited by in corpus (5)

• Survey on reinforcement learning for language processing
• Intelligent problem-solving as integrated hierarchical reinforcement learning
• An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey
• Hierarchical principles of embodied reinforcement learning: A review
• GRIMGEP: Learning Progress for Robust Goal Sampling in Visual Deep Reinforcement Learning