MapGo: Model-Assisted Policy Optimization for Goal-Oriented Tasks
arXiv:2105.06350
Abstract
In Goal-oriented Reinforcement learning, relabeling the raw goals in past experience to provide agents with hindsight ability is a major solution to the reward sparsity problem. In this paper, to enhance the diversity of relabeled goals, we develop FGI (Foresight Goal Inference), a new relabeling strategy that relabels the goals by looking into the future with a learned dynamics model. Besides, to improve sample efficiency, we propose to use the dynamics model to generate simulated trajectories for policy training. By integrating these two improvements, we introduce the MapGo framework (Model-Assisted Policy Optimization for Goal-oriented tasks). In our experiments, we first show the effectiveness of the FGI strategy compared with the hindsight one, and then show that the MapGo framework achieves higher sample efficiency when compared to model-free baselines on a set of complicated tasks.
arXiv commentPaper accepted at IJCAI 2021
References in corpus (9)
- Continuous control with deep reinforcement learning
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- Benchmarking Model-Based Reinforcement Learning
- Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
- Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
- Exploring Model-based Planning with Policy Networks
- Goal-Aware Prediction: Learning to Model What Matters
- Keeping Your Distance: Solving Sparse Reward Tasks Using Self-Balancing Shaped Rewards
- PlanGAN: Model-based Planning With Sparse Rewards and Multiple Goals