Towards Automatic Actor-Critic Solutions to Continuous Control

Abstract

Model-free off-policy actor-critic methods are an efficient solution to complex continuous control tasks. However, these algorithms rely on a number of design tricks and hyperparameters, making their application to new domains difficult and computationally expensive. This paper creates an evolutionary approach that automatically tunes these design decisions and eliminates the RL-specific hyperparameters from the Soft Actor-Critic algorithm. Our design is sample efficient and provides practical advantages over baseline approaches, including improved exploration, generalization over multiple control frequencies, and a robust ensemble of high-performance policies. Empirically, we show that our agent outperforms well-tuned hyperparameter settings in popular benchmarks from the DeepMind Control Suite. We then apply it to less common control tasks outside of simulated robotics to find high-performance solutions with minimal compute and research effort.

NeurIPS Deep RL Workshop 2021

References in corpus (10)

• Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
• DeepMind Control Suite
• If MaxEnt RL is the Answer, What is the Question?
• NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
• ORL: Reinforcement Learning Benchmarks for Online Stochastic Optimization Problems
• On Inductive Biases in Deep Reinforcement Learning
• Meta-SAC: Auto-tune the Entropy Temperature of Soft Actor-Critic via Metagradient
• An Analysis of Frame-skipping in Reinforcement Learning
• Online Hyper-parameter Tuning in Off-policy Learning via Evolutionary Strategies
• Online Meta-learning by Parallel Algorithm Competition