Exploring Model-based Planning with Policy Networks

Abstract

Model-based reinforcement learning (MBRL) with model-predictive control or online planning has shown great potential for locomotion control tasks in terms of both sample efficiency and asymptotic performance. Despite their initial successes, the existing planning methods search from candidate sequences randomly generated in the action space, which is inefficient in complex high-dimensional environments. In this paper, we propose a novel MBRL algorithm, model-based policy planning (POPLIN), that combines policy networks with online planning. More specifically, we formulate action planning at each time-step as an optimization problem using neural networks. We experiment with both optimization w.r.t. the action sequences initialized from the policy network, and also online optimization directly w.r.t. the parameters of the policy network. We show that POPLIN obtains state-of-the-art performance in the MuJoCo benchmarking environments, being about 3x more sample efficient than the state-of-the-art algorithms, such as PETS, TD3 and SAC. To explain the effectiveness of our algorithm, we show that the optimization surface in parameter space is smoother than in action space. Further more, we found the distilled policy network can be effectively applied without the expansive model predictive control during test time for some environments such as Cheetah. Code is released in https://github.com/WilsonWangTHU/POPLIN.

8 pages, 7 figures

References in corpus (13)

• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• Addressing Function Approximation Error in Actor-Critic Methods
• Emergence of Locomotion Behaviours in Rich Environments
• Recurrent World Models Facilitate Policy Evolution
• Continuous Deep Q-Learning with Model-based Acceleration
• Model-Ensemble Trust-Region Policy Optimization
• Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
• Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
• Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
• Combining Model-Based and Model-Free Updates for Trajectory-Centric Reinforcement Learning

Cited by in corpus (46)

• MOPO: Model-based Offline Policy Optimization
• Reinforcement Learning with Augmented Data
• MOReL : Model-Based Offline Reinforcement Learning
• Dream to Control: Learning Behaviors by Latent Imagination
• Deep Dynamics Models for Learning Dexterous Manipulation
• SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning
• Information Theoretic Regret Bounds for Online Nonlinear Control
• Model-based Reinforcement Learning for Semi-Markov Decision Processes with Neural ODEs
• Mastering Atari with Discrete World Models
• World Model as a Graph: Learning Latent Landmarks for Planning
• Objective Mismatch in Model-based Reinforcement Learning
• Model-Based Offline Planning
• Trust the Model When It Is Confident: Masked Model-based Actor-Critic
• Model-Based Visual Planning with Self-Supervised Functional Distances
• Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills
• On the model-based stochastic value gradient for continuous reinforcement learning
• Learning to Optimize in Model Predictive Control
• MBRL-Lib: A Modular Library for Model-based Reinforcement Learning
• Model-based Adversarial Meta-Reinforcement Learning
• Model-Predictive Control via Cross-Entropy and Gradient-Based Optimization
• Reset-Free Lifelong Learning with Skill-Space Planning
• A Tutorial on Sparse Gaussian Processes and Variational Inference
• Neural Dynamical Systems: Balancing Structure and Flexibility in Physical Prediction
• Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planning
• Delay-Aware Model-Based Reinforcement Learning for Continuous Control
• Adaptive Online Planning for Continual Lifelong Learning
• The Differentiable Cross-Entropy Method
• Scalable Online Planning via Reinforcement Learning Fine-Tuning
• Model-free and Bayesian Ensembling Model-based Deep Reinforcement Learning for Particle Accelerator Control Demonstrated on the FERMI FEL
• The LoCA Regret: A Consistent Metric to Evaluate Model-Based Behavior in Reinforcement Learning
• Control as Hybrid Inference
• Automating Experimental Optics with Sample Efficient Machine Learning Methods
• Model-Invariant State Abstractions for Model-Based Reinforcement Learning
• PerSim: Data-Efficient Offline Reinforcement Learning with Heterogeneous Agents via Personalized Simulators
• MapGo: Model-Assisted Policy Optimization for Goal-Oriented Tasks
• Learning Space Partitions for Path Planning
• Measuring Progress in Deep Reinforcement Learning Sample Efficiency
• On the Expressivity of Neural Networks for Deep Reinforcement Learning
• Model-based Meta Reinforcement Learning using Graph Structured Surrogate Models
• Bellman: A Toolbox for Model-Based Reinforcement Learning in TensorFlow
• Learning Pessimism for Robust and Efficient Off-Policy Reinforcement Learning
• Learning Off-Policy with Online Planning
• Planning with Exploration: Addressing Dynamics Bottleneck in Model-based Reinforcement Learning
• Discriminator Augmented Model-Based Reinforcement Learning
• Learning to Reweight Imaginary Transitions for Model-Based Reinforcement Learning
• Model Predictive Actor-Critic: Accelerating Robot Skill Acquisition with Deep Reinforcement Learning