Cautious Actor-Critic

Abstract

The oscillating performance of off-policy learning and persisting errors in the actor-critic (AC) setting call for algorithms that can conservatively learn to suit the stability-critical applications better. In this paper, we propose a novel off-policy AC algorithm cautious actor-critic (CAC). The name cautious comes from the doubly conservative nature that we exploit the classic policy interpolation from conservative policy iteration for the actor and the entropy-regularization of conservative value iteration for the critic. Our key observation is the entropy-regularized critic facilitates and simplifies the unwieldy interpolated actor update while still ensuring robust policy improvement. We compare CAC to state-of-the-art AC methods on a set of challenging continuous control problems and demonstrate that CAC achieves comparable performance while significantly stabilizes learning.

arXiv commentAccepted by Asian Conference on Machine Learning (ACML) 2021 as long oral presentation

References in corpus (8)

• Continuous control with deep reinforcement learning
• Trust Region Policy Optimization
• Reinforcement Learning with Deep Energy-Based Policies
• Bridging the Gap Between Value and Policy Based Reinforcement Learning
• Dream to Control: Learning Behaviors by Latent Imagination
• A Theory of Regularized Markov Decision Processes
• Diagnosing Bottlenecks in Deep Q-learning Algorithms
• Ensuring Monotonic Policy Improvement in Entropy-regularized Value-based Reinforcement Learning