Meta-SAC: Auto-tune the Entropy Temperature of Soft Actor-Critic via Metagradient

Abstract

Exploration-exploitation dilemma has long been a crucial issue in reinforcement learning. In this paper, we propose a new approach to automatically balance between these two. Our method is built upon the Soft Actor-Critic (SAC) algorithm, which uses an "entropy temperature" that balances the original task reward and the policy entropy, and hence controls the trade-off between exploitation and exploration. It is empirically shown that SAC is very sensitive to this hyperparameter, and the follow-up work (SAC-v2), which uses constrained optimization for automatic adjustment, has some limitations. The core of our method, namely Meta-SAC, is to use metagradient along with a novel meta objective to automatically tune the entropy temperature in SAC. We show that Meta-SAC achieves promising performances on several of the Mujoco benchmarking tasks, and outperforms SAC-v2 over 10% in one of the most challenging tasks, humanoid-v2.

arXiv commentpublished at 7th ICML Workshop on Automated Machine Learning (2020)

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Continuous control with deep reinforcement learning
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• Addressing Function Approximation Error in Actor-Critic Methods
• Soft Actor-Critic Algorithms and Applications
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• A Self-Tuning Actor-Critic Algorithm

Cited by in corpus (3)

• Towards Automatic Actor-Critic Solutions to Continuous Control
• Self-Imitation Learning by Planning
• Distilling Motion Planner Augmented Policies into Visual Control Policies for Robot Manipulation