Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errors

Abstract

In reinforcement learning (RL), function approximation errors are known to easily lead to the Q-value overestimations, thus greatly reducing policy performance. This paper presents a distributional soft actor-critic (DSAC) algorithm, which is an off-policy RL method for continuous control setting, to improve the policy performance by mitigating Q-value overestimations. We first discover in theory that learning a distribution function of state-action returns can effectively mitigate Q-value overestimations because it is capable of adaptively adjusting the update stepsize of the Q-value function. Then, a distributional soft policy iteration (DSPI) framework is developed by embedding the return distribution function into maximum entropy RL. Finally, we present a deep off-policy actor-critic variant of DSPI, called DSAC, which directly learns a continuous return distribution by keeping the variance of the state-action returns within a reasonable range to address exploding and vanishing gradient problems. We evaluate DSAC on the suite of MuJoCo continuous control tasks, achieving the state-of-the-art performance.

References in corpus (4)

• Emergence of Locomotion Behaviours in Rich Environments
• Hierarchical Reinforcement Learning for Self-Driving Decision-Making without Reliance on Labeled Driving Data
• Distributional Reinforcement Learning with Quantile Regression
• A Comparative Analysis of Expected and Distributional Reinforcement Learning

Cited by in corpus (20)

• Distributional Soft Actor-Critic with Three Refinements
• Adaptive dynamic programming for nonaffine nonlinear optimal control problem with state constraints
• A Survey on Reinforcement Learning in Aviation Applications
• Relaxed Actor-Critic with Convergence Guarantees for Continuous-Time Optimal Control of Nonlinear Systems
• On the Optimization Landscape of Dynamic Output Feedback Linear Quadratic Control
• Learning to be Safe: Deep RL with a Safety Critic
• Zero-touch Continuous Network Slicing Control via Scalable Actor-Critic Learning
• Fixed-Dimensional and Permutation Invariant State Representation of Autonomous Driving
• Optimization Landscape of Policy Gradient Methods for Discrete-time Static Output Feedback
• Optimization Landscape of Gradient Descent for Discrete-time Static Output Feedback
• Improving Generalization of Reinforcement Learning with Minimax Distributional Soft Actor-Critic
• Safe Reinforcement Learning for Autonomous Vehicles through Parallel Constrained Policy Optimization
• Conformal Symplectic Optimization for Stable Reinforcement Learning
• Encoding Distributional Soft Actor-Critic for Autonomous Driving in Multi-lane Scenarios
• Distributional Actor-Critic Ensemble for Uncertainty-Aware Continuous Control
• A Risk-Sensitive Approach to Policy Optimization
• On the Optimization Landscape of Dynamic Output Feedback: A Case Study for Linear Quadratic Regulator
• Mixed Reinforcement Learning with Additive Stochastic Uncertainty
• Progressive-Resolution Policy Distillation: Leveraging Coarse-Resolution Simulations for Time-Efficient Fine-Resolution Policy Learning
• GMAC: A Distributional Perspective on Actor-Critic Framework