Band-limited Soft Actor Critic Model
arXiv:2006.11431
Abstract
Soft Actor Critic (SAC) algorithms show remarkable performance in complex simulated environments. A key element of SAC networks is entropy regularization, which prevents the SAC actor from optimizing against fine grained features, oftentimes transient, of the state-action value function. This results in better sample efficiency during early training. We take this idea one step further by artificially bandlimiting the target critic spatial resolution through the addition of a convolutional filter. We derive the closed form solution in the linear case and show that bandlimiting reduces the interdependency between the low and high frequency components of the state-action value approximation, allowing the critic to learn faster. In experiments, the bandlimited SAC outperformed the classic twin-critic SAC in a number of Gym environments, and displayed more stability in returns. We derive novel insights about SAC by adding a stochastic noise disturbance, a technique that is increasingly being used to learn robust policies that transfer well to the real world counterparts.
arXiv comment8 pages plus additional material
References in corpus (9)
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Addressing Function Approximation Error in Actor-Critic Methods
- Benchmarking Deep Reinforcement Learning for Continuous Control
- Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
- On the Spectral Bias of Neural Networks
- Band-limited Training and Inference for Convolutional Neural Networks
- Leveraging exploration in off-policy algorithms via normalizing flows
- Smoothed Action Value Functions for Learning Gaussian Policies
- A Convergence Result for Regularized Actor-Critic Methods