Model-Based Reinforcement Learning via Meta-Policy Optimization

Abstract

Model-based reinforcement learning approaches carry the promise of being data efficient. However, due to challenges in learning dynamics models that sufficiently match the real-world dynamics, they struggle to achieve the same asymptotic performance as model-free methods. We propose Model-Based Meta-Policy-Optimization (MB-MPO), an approach that foregoes the strong reliance on accurate learned dynamics models. Using an ensemble of learned dynamic models, MB-MPO meta-learns a policy that can quickly adapt to any model in the ensemble with one policy gradient step. This steers the meta-policy towards internalizing consistent dynamics predictions among the ensemble while shifting the burden of behaving optimally w.r.t. the model discrepancies towards the adaptation step. Our experiments show that MB-MPO is more robust to model imperfections than previous model-based approaches. Finally, we demonstrate that our approach is able to match the asymptotic performance of model-free methods while requiring significantly less experience.

arXiv commentFirst 2 authors contributed equally. Accepted for Conference on Robot Learning (CoRL)

Cited by in corpus (55)

• MOPO: Model-based Offline Policy Optimization
• Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
• When to Trust Your Model: Model-Based Policy Optimization
• Learning Scheduling Algorithms for Data Processing Clusters
• Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization
• Variational Inference MPC for Bayesian Model-based Reinforcement Learning
• Data Efficient Reinforcement Learning for Legged Robots
• Deep Reinforcement Learning in Computer Vision: A Comprehensive Survey
• Learning latent state representation for speeding up exploration
• Robust Model-based Reinforcement Learning for Autonomous Greenhouse Control
• Modelling Generalized Forces with Reinforcement Learning for Sim-to-Real Transfer
• Asynchronous Methods for Model-Based Reinforcement Learning
• Dynamics-Aware Quality-Diversity for Efficient Learning of Skill Repertoires
• ModelLight: Model-Based Meta-Reinforcement Learning for Traffic Signal Control
• The Gap Between Model-Based and Model-Free Methods on the Linear Quadratic Regulator: An Asymptotic Viewpoint
• Combining Model-Based and Model-Free Methods for Nonlinear Control: A Provably Convergent Policy Gradient Approach
• Uncertainty-aware Model-based Policy Optimization
• Offline Meta Learning of Exploration
• PlaNet of the Bayesians: Reconsidering and Improving Deep Planning Network by Incorporating Bayesian Inference
• Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environment
• Model-based Lookahead Reinforcement Learning
• DyNODE: Neural Ordinary Differential Equations for Dynamics Modeling in Continuous Control
• Adaptive Online Planning for Continual Lifelong Learning
• Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planning
• Ready Policy One: World Building Through Active Learning
• OffWorld Gym: open-access physical robotics environment for real-world reinforcement learning benchmark and research
• Deep Residual Reinforcement Learning
• Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling
• Trajectory-wise Multiple Choice Learning for Dynamics Generalization in Reinforcement Learning
• Regularizing Trajectory Optimization with Denoising Autoencoders
• Imagined Value Gradients: Model-Based Policy Optimization with Transferable Latent Dynamics Models
• Model-based Policy Optimization with Unsupervised Model Adaptation
• Model Imitation for Model-Based Reinforcement Learning
• Meta-Model-Based Meta-Policy Optimization
• RLlib Flow: Distributed Reinforcement Learning is a Dataflow Problem
• Multi-Agent Reinforcement Learning for Unmanned Aerial Vehicle Coordination by Multi-Critic Policy Gradient Optimization
• Regularizing Model-Based Planning with Energy-Based Models
• On Effective Scheduling of Model-based Reinforcement Learning
• VMAV-C: A Deep Attention-based Reinforcement Learning Algorithm for Model-based Control
• Generalized Hidden Parameter MDPs Transferable Model-based RL in a Handful of Trials
• PerSim: Data-Efficient Offline Reinforcement Learning with Heterogeneous Agents via Personalized Simulators
• Imaginary Hindsight Experience Replay: Curious Model-based Learning for Sparse Reward Tasks
• Learning to Plan Optimistically: Uncertainty-Guided Deep Exploration via Latent Model Ensembles
• Measuring Progress in Deep Reinforcement Learning Sample Efficiency
• Neural-iLQR: A Learning-Aided Shooting Method for Trajectory Optimization
• Towards Tractable Optimism in Model-Based Reinforcement Learning
• A Contraction Approach to Model-based Reinforcement Learning
• Planning with Exploration: Addressing Dynamics Bottleneck in Model-based Reinforcement Learning
• Model Embedding Model-Based Reinforcement Learning
• Model Predictive Actor-Critic: Accelerating Robot Skill Acquisition with Deep Reinforcement Learning
• Dynamic Horizon Value Estimation for Model-based Reinforcement Learning
• Improving Hyperparameter Optimization by Planning Ahead
• Model-Augmented Q-learning
• Minimax Model Learning
• Learning to Reweight Imaginary Transitions for Model-Based Reinforcement Learning