A Self-Tuning Actor-Critic Algorithm

Abstract

Reinforcement learning algorithms are highly sensitive to the choice of hyperparameters, typically requiring significant manual effort to identify hyperparameters that perform well on a new domain. In this paper, we take a step towards addressing this issue by using metagradients to automatically adapt hyperparameters online by meta-gradient descent (Xu et al., 2018). We apply our algorithm, Self-Tuning Actor-Critic (STAC), to self-tune all the differentiable hyperparameters of an actor-critic loss function, to discover auxiliary tasks, and to improve off-policy learning using a novel leaky V-trace operator. STAC is simple to use, sample efficient and does not require a significant increase in compute. Ablative studies show that the overall performance of STAC improved as we adapt more hyperparameters. When applied to the Arcade Learning Environment (Bellemare et al. 2012), STAC improved the median human normalized score in 200M steps from 243% to 364%. When applied to the DM Control suite (Tassa et al., 2018), STAC improved the mean score in 30M steps from 217 to 389 when learning with features, from 108 to 202 when learning from pixels, and from 195 to 295 in the Real-World Reinforcement Learning Challenge (Dulac-Arnold et al., 2020).

References in corpus (17)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Empirical Evaluation of Rectified Activations in Convolutional Network
• Soft Actor-Critic Algorithms and Applications
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• DeepMind Control Suite
• Meta-Gradient Reinforcement Learning
• Hyperparameter optimization with approximate gradient
• Hyperbolic Discounting and Learning over Multiple Horizons
• Forward and Reverse Gradient-Based Hyperparameter Optimization
• An empirical investigation of the challenges of real-world reinforcement learning
• Discovery of Useful Questions as Auxiliary Tasks
• Fast Efficient Hyperparameter Tuning for Policy Gradients
• Podracer architectures for scalable Reinforcement Learning
• Adaptive Lambda Least-Squares Temporal Difference Learning
• Adaptive Trade-Offs in Off-Policy Learning
• Adapting Behaviour for Learning Progress
• Online Hyper-parameter Tuning in Off-policy Learning via Evolutionary Strategies

Cited by in corpus (15)

• Evolutionary Reinforcement Learning: A Survey
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Discovering Reinforcement Learning Algorithms
• Meta-Gradient Reinforcement Learning with an Objective Discovered Online
• Meta-SAC: Auto-tune the Entropy Temperature of Soft Actor-Critic via Metagradient
• Hyperparameter Auto-tuning in Self-Supervised Robotic Learning
• Podracer architectures for scalable Reinforcement Learning
• Temporal Difference Uncertainties as a Signal for Exploration
• Online Hyper-parameter Tuning in Off-policy Learning via Evolutionary Strategies
• Discovery of Options via Meta-Learned Subgoals
• Towards Automatic Actor-Critic Solutions to Continuous Control
• Unifying Gradient Estimators for Meta-Reinforcement Learning via Off-Policy Evaluation
• Neural Auto-Curricula
• Bootstrapped Meta-Learning
• One Step at a Time: Pros and Cons of Multi-Step Meta-Gradient Reinforcement Learning