PAC-Bayesian Soft Actor-Critic Learning

Abstract

Actor-critic algorithms address the dual goals of reinforcement learning (RL), policy evaluation and improvement via two separate function approximators. The practicality of this approach comes at the expense of training instability, caused mainly by the destructive effect of the approximation errors of the critic on the actor. We tackle this bottleneck by employing an existing Probably Approximately Correct (PAC) Bayesian bound for the first time as the critic training objective of the Soft Actor-Critic (SAC) algorithm. We further demonstrate that online learning performance improves significantly when a stochastic actor explores multiple futures by critic-guided random search. We observe our resulting algorithm to compare favorably against the state-of-the-art SAC implementation on multiple classical control and locomotion tasks in terms of both sample efficiency and regret.

arXiv comment19 pages, 2 figures

References in corpus (14)

• Adam: A Method for Stochastic Optimization
• Denoising Diffusion Probabilistic Models
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• Addressing Function Approximation Error in Actor-Critic Methods
• OpenAI Gym
• Maximum Entropy Deep Inverse Reinforcement Learning
• Softmax Deep Double Deterministic Policy Gradients
• Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learning
• Tactical Optimism and Pessimism for Deep Reinforcement Learning
• Second Order PAC-Bayesian Bounds for the Weighted Majority Vote
• Diversity Actor-Critic: Sample-Aware Entropy Regularization for Sample-Efficient Exploration
• Robust Generalised Bayesian Inference for Intractable Likelihoods
• PAC-Bayesian Policy Evaluation for Reinforcement Learning