Reinforcement Learning by Value Gradients

Abstract

The concept of the value-gradient is introduced and developed in the context of reinforcement learning. It is shown that by learning the value-gradients exploration or stochastic behaviour is no longer needed to find locally optimal trajectories. This is the main motivation for using value-gradients, and it is argued that learning value-gradients is the actual objective of any value-function learning algorithm for control problems. It is also argued that learning value-gradients is significantly more efficient than learning just the values, and this argument is supported in experiments by efficiency gains of several orders of magnitude, in several problem domains. Once value-gradients are introduced into learning, several analyses become possible. For example, a surprising equivalence between a value-gradient learning algorithm and a policy-gradient learning algorithm is proven, and this provides a robust convergence proof for control problems using a value function with a general function approximator.

References in corpus (1)

• Stable Adaptive Control Using New Critic Designs

Cited by in corpus (9)

• Learning Implicit Credit Assignment for Cooperative Multi-Agent Reinforcement Learning
• On the role of planning in model-based deep reinforcement learning
• How to Learn a Useful Critic? Model-based Action-Gradient-Estimator Policy Optimization
• The Local Optimality of Reinforcement Learning by Value Gradients, and its Relationship to Policy Gradient Learning
• Bridging Imagination and Reality for Model-Based Deep Reinforcement Learning
• The Importance of Clipping in Neurocontrol by Direct Gradient Descent on the Cost-to-Go Function and in Adaptive Dynamic Programming
• Deterministic Policy Gradients With General State Transitions
• The Divergence of Reinforcement Learning Algorithms with Value-Iteration and Function Approximation
• The Gambler's Problem and Beyond