How to Learn a Useful Critic? Model-based Action-Gradient-Estimator Policy Optimization

Abstract

Deterministic-policy actor-critic algorithms for continuous control improve the actor by plugging its actions into the critic and ascending the action-value gradient, which is obtained by chaining the actor's Jacobian matrix with the gradient of the critic with respect to input actions. However, instead of gradients, the critic is, typically, only trained to accurately predict expected returns, which, on their own, are useless for policy optimization. In this paper, we propose MAGE, a model-based actor-critic algorithm, grounded in the theory of policy gradients, which explicitly learns the action-value gradient. MAGE backpropagates through the learned dynamics to compute gradient targets in temporal difference learning, leading to a critic tailored for policy improvement. On a set of MuJoCo continuous-control tasks, we demonstrate the efficiency of the algorithm in comparison to model-free and model-based state-of-the-art baselines.

References in corpus (23)

• Deep Learning in Neural Networks: An Overview
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Continuous control with deep reinforcement learning
• Trust Region Policy Optimization
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Searching for Activation Functions
• On the Variance of the Adaptive Learning Rate and Beyond
• Learning Continuous Control Policies by Stochastic Value Gradients
• Model-Ensemble Trust-Region Policy Optimization
• Gradient Estimation Using Stochastic Computation Graphs
• Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
• When to use parametric models in reinforcement learning?
• Model-Based Active Exploration
• DeepMDP: Learning Continuous Latent Space Models for Representation Learning
• First-order Adversarial Vulnerability of Neural Networks and Input Dimension
• Model-Augmented Actor-Critic: Backpropagating through Paths
• A Closer Look at Deep Policy Gradients
• Compatible Value Gradients for Reinforcement Learning of Continuous Deep Policies
• Reinforcement Learning by Value Gradients
• Policy Optimization via Importance Sampling
• Credit Assignment Techniques in Stochastic Computation Graphs
• Gradient-Aware Model-based Policy Search
• On approximating $\nabla f$ with neural networks

Cited by in corpus (3)

• Machine learning for industrial sensing and control: A survey and practical perspective
• A case for new neural network smoothness constraints
• A Unified Off-Policy Evaluation Approach for General Value Function