Curiosity-Driven Multi-Criteria Hindsight Experience Replay

Abstract

Dealing with sparse rewards is a longstanding challenge in reinforcement learning. The recent use of hindsight methods have achieved success on a variety of sparse-reward tasks, but they fail on complex tasks such as stacking multiple blocks with a robot arm in simulation. Curiosity-driven exploration using the prediction error of a learned dynamics model as an intrinsic reward has been shown to be effective for exploring a number of sparse-reward environments. We present a method that combines hindsight with curiosity-driven exploration and curriculum learning in order to solve the challenging sparse-reward block stacking task. We are the first to stack more than two blocks using only sparse reward without human demonstrations.

14 pages

References in corpus (18)

• Continuous control with deep reinforcement learning
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• Parameter Space Noise for Exploration
• Large-Scale Study of Curiosity-Driven Learning
• Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
• Exploration by Random Network Distillation
• Go-Explore: a New Approach for Hard-Exploration Problems
• Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research
• Variational Intrinsic Control
• Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning
• Reverse Curriculum Generation for Reinforcement Learning
• Data-efficient Deep Reinforcement Learning for Dexterous Manipulation
• Curiosity Driven Exploration of Learned Disentangled Goal Spaces
• Unsupervised Learning of Goal Spaces for Intrinsically Motivated Goal Exploration
• Curiosity-Driven Experience Prioritization via Density Estimation
• Energy-Based Hindsight Experience Prioritization
• Accuracy-based Curriculum Learning in Deep Reinforcement Learning
• Improvements on Hindsight Learning

Cited by in corpus (5)

• Grounding Language to Autonomously-Acquired Skills via Goal Generation
• Effects of sparse rewards of different magnitudes in the speed of learning of model-based actor critic methods
• Imaginary Hindsight Experience Replay: Curious Model-based Learning for Sparse Reward Tasks
• Learning Compositional Neural Programs for Continuous Control
• Automatic Curricula via Expert Demonstrations