TD-Regularized Actor-Critic Methods

Abstract

Actor-critic methods can achieve incredible performance on difficult reinforcement learning problems, but they are also prone to instability. This is partly due to the interaction between the actor and critic during learning, e.g., an inaccurate step taken by one of them might adversely affect the other and destabilize the learning. To avoid such issues, we propose to regularize the learning objective of the actor by penalizing the temporal difference (TD) error of the critic. This improves stability by avoiding large steps in the actor update whenever the critic is highly inaccurate. The resulting method, which we call the TD-regularized actor-critic method, is a simple plug-and-play approach to improve stability and overall performance of the actor-critic methods. Evaluations on standard benchmarks confirm this.

References in corpus (12)

• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Addressing Function Approximation Error in Actor-Critic Methods
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Asynchronous Methods for Deep Reinforcement Learning
• Policy Gradients with Variance Related Risk Criteria
• Safe and Efficient Off-Policy Reinforcement Learning
• Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines
• MuProp: Unbiased Backpropagation for Stochastic Neural Networks
• Trust-PCL: An Off-Policy Trust Region Method for Continuous Control
• Towards Generalization and Simplicity in Continuous Control
• The Reactor: A fast and sample-efficient Actor-Critic agent for Reinforcement Learning

Cited by in corpus (6)

• Forward and inverse reinforcement learning sharing network weights and hyperparameters
• Proximal Policy Optimization with Relative Pearson Divergence
• Reinforcement Learning Your Way: Agent Characterization through Policy Regularization
• Reward Bonuses with Gain Scheduling Inspired by Iterative Deepening Search
• Design of Restricted Normalizing Flow towards Arbitrary Stochastic Policy with Computational Efficiency
• Towards Deep Robot Learning with Optimizer applicable to Non-stationary Problems