Policy-Aware Model Learning for Policy Gradient Methods
arXiv:2003.00030
Abstract
This paper considers the problem of learning a model in model-based reinforcement learning (MBRL). We examine how the planning module of an MBRL algorithm uses the model, and propose that the model learning module should incorporate the way the planner is going to use the model. This is in contrast to conventional model learning approaches, such as those based on maximum likelihood estimate, that learn a predictive model of the environment without explicitly considering the interaction of the model and the planner. We focus on policy gradient type of planning algorithms and derive new loss functions for model learning that incorporate how the planner uses the model. We call this approach Policy-Aware Model Learning (PAML). We theoretically analyze a generic model-based policy gradient algorithm and provide a convergence guarantee for the optimized policy. We also empirically evaluate PAML on some benchmark problems, showing promising results.
References in corpus (16)
- Continuous control with deep reinforcement learning
- Trust Region Policy Optimization
- Addressing Function Approximation Error in Actor-Critic Methods
- Recurrent World Models Facilitate Policy Evolution
- Model-Ensemble Trust-Region Policy Optimization
- Benchmarking Model-Based Reinforcement Learning
- The Predictron: End-To-End Learning and Planning
- Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
- On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift
- Dyna-Style Planning with Linear Function Approximation and Prioritized Sweeping
- Neural Policy Gradient Methods: Global Optimality and Rates of Convergence
- Neural Proximal/Trust Region Policy Optimization Attains Globally Optimal Policy
- Information-Theoretic Considerations in Batch Reinforcement Learning
- Self-Correcting Models for Model-Based Reinforcement Learning
- Gradient-Aware Model-based Policy Search
- Equivalence Between Wasserstein and Value-Aware Loss for Model-based Reinforcement Learning