Towards Automatic Actor-Critic Solutions to Continuous Control

Abstract

Model-free off-policy actor-critic methods are an efficient solution to complex continuous control tasks. However, these algorithms rely on a number of design tricks and hyperparameters, making their application to new domains difficult and computationally expensive. This paper creates an evolutionary approach that automatically tunes these design decisions and eliminates the RL-specific hyperparameters from the Soft Actor-Critic algorithm. Our design is sample efficient and provides practical advantages over baseline approaches, including improved exploration, generalization over multiple control frequencies, and a robust ensemble of high-performance policies. Empirically, we show that our agent outperforms well-tuned hyperparameter settings in popular benchmarks from the DeepMind Control Suite. We then apply it to less common control tasks outside of simulated robotics to find high-performance solutions with minimal compute and research effort.

arXiv commentNeurIPS Deep RL Workshop 2021

References in corpus (30)

• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Addressing Function Approximation Error in Actor-Critic Methods
• Soft Actor-Critic Algorithms and Applications
• AutoML: A Survey of the State-of-the-Art
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• DeepMind Control Suite
• Distributed Distributional Deterministic Policy Gradients
• Exploration by Random Network Distillation
• Agent57: Outperforming the Atari Human Benchmark
• SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning
• Learning to Locomote: Understanding How Environment Design Matters for Deep Reinforcement Learning
• DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction
• A Self-Tuning Actor-Critic Algorithm
• If MaxEnt RL is the Answer, What is the Question?
• NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
• ORL: Reinforcement Learning Benchmarks for Online Stochastic Optimization Problems
• On Inductive Biases in Deep Reinforcement Learning
• Meta-SAC: Auto-tune the Entropy Temperature of Soft Actor-Critic via Metagradient
• Fast Efficient Hyperparameter Tuning for Policy Gradients
• An Analysis of Frame-skipping in Reinforcement Learning
• TAAC: Temporally Abstract Actor-Critic for Continuous Control
• Online Hyper-parameter Tuning in Off-policy Learning via Evolutionary Strategies
• GRAC: Self-Guided and Self-Regularized Actor-Critic
• Introduction to the "Industrial Benchmark"
• Online Meta-learning by Parallel Algorithm Competition

Cited by in corpus (1)

• TAAC: Temporally Abstract Actor-Critic for Continuous Control