Model-Augmented Actor-Critic: Backpropagating through Paths

Abstract

Current model-based reinforcement learning approaches use the model simply as a learned black-box simulator to augment the data for policy optimization or value function learning. In this paper, we show how to make more effective use of the model by exploiting its differentiability. We construct a policy optimization algorithm that uses the pathwise derivative of the learned model and policy across future timesteps. Instabilities of learning across many timesteps are prevented by using a terminal value function, learning the policy in an actor-critic fashion. Furthermore, we present a derivation on the monotonic improvement of our objective in terms of the gradient error in the model and value function. We show that our approach (i) is consistently more sample efficient than existing state-of-the-art model-based algorithms, (ii) matches the asymptotic performance of model-free algorithms, and (iii) scales to long horizons, a regime where typically past model-based approaches have struggled.

Accepted paper at ICLR 2020

References in corpus (15)

• Auto-Encoding Variational Bayes
• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• Addressing Function Approximation Error in Actor-Critic Methods
• Soft Actor-Critic Algorithms and Applications
• Model-Ensemble Trust-Region Policy Optimization
• Benchmarking Model-Based Reinforcement Learning
• Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
• Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion
• Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
• When to Trust Your Model: Model-Based Policy Optimization
• Universal Planning Networks
• Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control
• MPC-Inspired Neural Network Policies for Sequential Decision Making

Cited by in corpus (19)

• The Free Energy Principle for Perception and Action: A Deep Learning Perspective
• Randomized Ensembled Double Q-Learning: Learning Fast Without a Model
• Model-based Reinforcement Learning for Semi-Markov Decision Processes with Neural ODEs
• Optimistic Reinforcement Learning by Forward Kullback-Leibler Divergence Optimization
• On the model-based stochastic value gradient for continuous reinforcement learning
• How to Learn a Useful Critic? Model-based Action-Gradient-Estimator Policy Optimization
• Adaptive and Multiple Time-scale Eligibility Traces for Online Deep Reinforcement Learning
• Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planning
• Provable Model-based Nonlinear Bandit and Reinforcement Learning: Shelve Optimism, Embrace Virtual Curvature
• Optimization Algorithm for Feedback and Feedforward Policies towards Robot Control Robust to Sensing Failures
• Iterative Amortized Policy Optimization
• On Effective Scheduling of Model-based Reinforcement Learning
• On the Expressivity of Neural Networks for Deep Reinforcement Learning
• Model-based Multi-agent Policy Optimization with Adaptive Opponent-wise Rollouts
• Learning Off-Policy with Online Planning
• Model Predictive Actor-Critic: Accelerating Robot Skill Acquisition with Deep Reinforcement Learning
• On-Policy Model Errors in Reinforcement Learning
• Contrastive Active Inference
• OffCon${}^3$: What is state of the art anyway?