A Game Theoretic Framework for Model Based Reinforcement Learning

Abstract

Model-based reinforcement learning (MBRL) has recently gained immense interest due to its potential for sample efficiency and ability to incorporate off-policy data. However, designing stable and efficient MBRL algorithms using rich function approximators have remained challenging. To help expose the practical challenges in MBRL and simplify algorithm design from the lens of abstraction, we develop a new framework that casts MBRL as a game between: (1) a policy player, which attempts to maximize rewards under the learned model; (2) a model player, which attempts to fit the real-world data collected by the policy player. For algorithm development, we construct a Stackelberg game between the two players, and show that it can be solved with approximate bi-level optimization. This gives rise to two natural families of algorithms for MBRL based on which player is chosen as the leader in the Stackelberg game. Together, they encapsulate, unify, and generalize many previous MBRL algorithms. Furthermore, our framework is consistent with and provides a clear basis for heuristics known to be important in practice from prior works. Finally, through experiments we validate that our proposed algorithms are highly sample efficient, match the asymptotic performance of model-free policy gradient, and scale gracefully to high-dimensional tasks like dexterous hand manipulation. Additional details and code can be obtained from the project page at https://sites.google.com/view/mbrl-game

arXiv commentICML 2020. This version contains expanded discussion, hyperparameter configurations, and ablation studies

References in corpus (19)

• Adam: A Method for Stochastic Optimization
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• On First-Order Meta-Learning Algorithms
• Unrolled Generative Adversarial Networks
• Model-Ensemble Trust-Region Policy Optimization
• Benchmarking Model-Based Reinforcement Learning
• Meta-Learning with Implicit Gradients
• Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
• Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
• When to Trust Your Model: Model-Based Policy Optimization
• Convergence rate of linear two-time-scale stochastic approximation
• Provably Efficient Maximum Entropy Exploration
• Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines
• Deep Dynamics Models for Learning Dexterous Manipulation
• Agnostic System Identification for Model-Based Reinforcement Learning
• Competitive Gradient Descent
• Convergence of Learning Dynamics in Stackelberg Games
• On Solving Minimax Optimization Locally: A Follow-the-Ridge Approach

Cited by in corpus (19)

• MOReL : Model-Based Offline Reinforcement Learning
• Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization
• Randomized Ensembled Double Q-Learning: Learning Fast Without a Model
• Model-Based Offline Planning
• Offline Reinforcement Learning from Images with Latent Space Models
• Model-based Adversarial Meta-Reinforcement Learning
• Dynamics-Aware Quality-Diversity for Efficient Learning of Skill Repertoires
• Solving Structured Hierarchical Games Using Differential Backward Induction
• Visual Adversarial Imitation Learning using Variational Models
• Meta-Model-Based Meta-Policy Optimization
• Iterative Amortized Policy Optimization
• Approximate Model-Based Shielding for Safe Reinforcement Learning
• On Effective Scheduling of Model-based Reinforcement Learning
• MobILE: Model-Based Imitation Learning From Observation Alone
• Model-based Meta Reinforcement Learning using Graph Structured Surrogate Models
• Learning Off-Policy with Online Planning
• Complex Momentum for Optimization in Games
• Model-Augmented Q-learning
• Minimax Model Learning