Imaginary Hindsight Experience Replay: Curious Model-based Learning for Sparse Reward Tasks
arXiv:2110.02414
Abstract
Model-based reinforcement learning is a promising learning strategy for practical robotic applications due to its improved data-efficiency versus model-free counterparts. However, current state-of-the-art model-based methods rely on shaped reward signals, which can be difficult to design and implement. To remedy this, we propose a simple model-based method tailored for sparse-reward multi-goal tasks that foregoes the need for complicated reward engineering. This approach, termed Imaginary Hindsight Experience Replay, minimises real-world interactions by incorporating imaginary data into policy updates. To improve exploration in the sparse-reward setting, the policy is trained with standard Hindsight Experience Replay and endowed with curiosity-based intrinsic rewards. Upon evaluation, this approach provides an order of magnitude increase in data-efficiency on average versus the state-of-the-art model-free method in the benchmark OpenAI Gym Fetch Robotics tasks.
References in corpus (28)
- Adam: A Method for Stochastic Optimization
- Continuous control with deep reinforcement learning
- Dota 2 with Large Scale Deep Reinforcement Learning
- Model-Based Reinforcement Learning for Atari
- Large-Scale Study of Curiosity-Driven Learning
- Learning Latent Dynamics for Planning from Pixels
- Hindsight Experience Replay
- Continuous Deep Q-Learning with Model-based Acceleration
- Visual Reinforcement Learning with Imagined Goals
- Model-Ensemble Trust-Region Policy Optimization
- Exploration by Random Network Distillation
- Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research
- A Deeper Look at Experience Replay
- Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
- Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
- Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
- Data-efficient Deep Reinforcement Learning for Dexterous Manipulation
- Model-Based Reinforcement Learning via Meta-Policy Optimization
- Revisiting Fundamentals of Experience Replay
- Model-Based Active Exploration
- Deep Dynamics Models for Learning Dexterous Manipulation
- PlanGAN: Model-based Planning With Sparse Rewards and Multiple Goals
- Explicit Explore-Exploit Algorithms in Continuous State Spaces
- MHER: Model-based Hindsight Experience Replay
- Soft Hindsight Experience Replay
- Multi-objective Model-based Policy Search for Data-efficient Learning with Sparse Rewards
- Curiosity-Driven Multi-Criteria Hindsight Experience Replay
- Episodic Self-Imitation Learning with Hindsight