Automated Reinforcement Learning (AutoRL): A Survey and Open Problems

Abstract

The combination of Reinforcement Learning (RL) with deep learning has led to a series of impressive feats, with many believing (deep) RL provides a path towards generally capable agents. However, the success of RL agents is often highly sensitive to design choices in the training process, which may require tedious and error-prone manual tuning. This makes it challenging to use RL for new problems, while also limits its full potential. In many other areas of machine learning, AutoML has shown it is possible to automate such design choices and has also yielded promising initial results when applied to RL. However, Automated Reinforcement Learning (AutoRL) involves not only standard applications of AutoML but also includes additional challenges unique to RL, that naturally produce a different set of methods. As such, AutoRL has been emerging as an important area of research in RL, providing promise in a variety of applications from RNA design to playing games such as Go. Given the diversity of methods and environments considered in RL, much of the research has been conducted in distinct subfields, ranging from meta-learning to evolution. In this survey we seek to unify the field of AutoRL, we provide a common taxonomy, discuss each area in detail and pose open problems which would be of interest to researchers going forward.

arXiv commentPublished in JAIR. Co-first authors and co-last authors are listed in alphabetical order

References in corpus (122)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Continuous control with deep reinforcement learning
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Trust Region Policy Optimization
• A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
• Prioritized Experience Replay
• Soft Actor-Critic Algorithms and Applications
• Neural Architecture Search: A Survey
• Dueling Network Architectures for Deep Reinforcement Learning
• Asynchronous Methods for Deep Reinforcement Learning
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization
• Dota 2 with Large Scale Deep Reinforcement Learning
• ParamILS: An Automatic Algorithm Configuration Framework
• Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
• Solving Rubik's Cube with a Robot Hand
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Noisy Networks for Exploration
• Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning
• Conservative Q-Learning for Offline Reinforcement Learning
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Recurrent World Models Facilitate Policy Evolution
• Model-Based Reinforcement Learning for Atari
• On Exact Computation with an Infinitely Wide Neural Net
• Learning to learn by gradient descent by gradient descent
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• Learning Dexterous In-Hand Manipulation
• Population Based Training of Neural Networks
• A Distributional Perspective on Reinforcement Learning
• Bayesian optimization for materials design
• Non-stochastic Best Arm Identification and Hyperparameter Optimization
• A Study on Overfitting in Deep Reinforcement Learning
• Agent57: Outperforming the Atari Human Benchmark
• Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
• How Does Batch Normalization Help Optimization?
• Hyperparameter Importance Across Datasets
• BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization
• Quantifying Generalization in Reinforcement Learning
• Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control
• BOHB: Robust and Efficient Hyperparameter Optimization at Scale
• Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
• Meta-Gradient Reinforcement Learning
• Learning to Learn without Gradient Descent by Gradient Descent
• Leveraging Procedural Generation to Benchmark Reinforcement Learning
• Automatic Goal Generation for Reinforcement Learning Agents
• Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets
• In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning
• Dream to Control: Learning Behaviors by Latent Imagination
• Evolved Policy Gradients
• Stabilizing Transformers for Reinforcement Learning
• Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play
• When to Trust Your Model: Model-Based Policy Optimization
• Sim-to-Real: Learning Agile Locomotion For Quadruped Robots
• A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation
• Learning to Utilize Shaping Rewards: A New Approach of Reward Shaping
• Theory of Parameter Control for Discrete Black-Box Optimization: Provable Performance Gains Through Dynamic Parameter Choices
• Emergent Coordination Through Competition
• Revisiting Fundamentals of Experience Replay
• Bayesian Optimization in AlphaGo
• Implicit Regularization in Deep Learning
• Parallel and Distributed Thompson Sampling for Large-scale Accelerated Exploration of Chemical Space
• VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning
• Discovering Reinforcement Learning Algorithms
• How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments
• Improving Generalization in Meta Reinforcement Learning using Learned Objectives
• Open-Ended Learning Leads to Generally Capable Agents
• Automatic Data Augmentation for Generalization in Deep Reinforcement Learning
• Learning with AMIGo: Adversarially Motivated Intrinsic Goals
• The NetHack Learning Environment
• The MineRL 2019 Competition on Sample Efficient Reinforcement Learning using Human Priors
• Effective Diversity in Population Based Reinforcement Learning
• Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions
• Reward Shaping via Meta-Learning
• Deep Reinforcement Learning at the Edge of the Statistical Precipice
• Safe Reinforcement Learning via Curriculum Induction
• Munchausen Reinforcement Learning
• Meta-Gradient Reinforcement Learning with an Objective Discovered Online
• Automatic Curriculum Learning through Value Disagreement
• On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learning
• A Self-Tuning Actor-Critic Algorithm
• Learning to Repeat: Fine Grained Action Repetition for Deep Reinforcement Learning
• On Learning Intrinsic Rewards for Policy Gradient Methods
• Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design
• Evolving Rewards to Automate Reinforcement Learning
• On Inductive Biases in Deep Reinforcement Learning
• Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation
• Asymmetric self-play for automatic goal discovery in robotic manipulation
• Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Research
• Evaluating the Performance of Reinforcement Learning Algorithms
• Observational Overfitting in Reinforcement Learning
• Think Global and Act Local: Bayesian Optimisation over High-Dimensional Categorical and Mixed Search Spaces
• Provably Efficient Online Hyperparameter Optimization with Population-Based Bandits
• D2RL: Deep Dense Architectures in Reinforcement Learning
• From Motor Control to Team Play in Simulated Humanoid Football
• Fast Efficient Hyperparameter Tuning for Policy Gradients
• MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Research
• Knowing The What But Not The Where in Bayesian Optimization
• Transient Non-Stationarity and Generalisation in Deep Reinforcement Learning
• Quantifying Differences in Reward Functions
• Adaptive Trade-Offs in Off-Policy Learning
• An Empirical Study on Hyperparameters and their Interdependence for RL Generalization
• A Greedy Approach to Adapting the Trace Parameter for Temporal Difference Learning
• Ready Policy One: World Building Through Active Learning
• Revisiting Design Choices in Offline Model-Based Reinforcement Learning
• TempoRL: Learning When to Act
• Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning
• Environment Generation for Zero-Shot Compositional Reinforcement Learning
• Discovery of Options via Meta-Learned Subgoals
• TeachMyAgent: a Benchmark for Automatic Curriculum Learning in Deep RL
• Quantity vs. Quality: On Hyperparameter Optimization for Deep Reinforcement Learning
• Sample-Efficient Automated Deep Reinforcement Learning
• Faster Improvement Rate Population Based Training
• Tuning Mixed Input Hyperparameters on the Fly for Efficient Population Based AutoRL
• Adaptive Temporal-Difference Learning for Policy Evaluation with Per-State Uncertainty Estimates
• CARL: A Benchmark for Contextual and Adaptive Reinforcement Learning
• Adaptive Procedural Task Generation for Hard-Exploration Problems
• Self-Paced Context Evaluation for Contextual Reinforcement Learning
• Learning Synthetic Environments for Reinforcement Learning with Evolution Strategies
• Hyperparameters in Contextual RL are Highly Situational
• The Cross-environment Hyperparameter Setting Benchmark for Reinforcement Learning

Cited by in corpus (5)

• Evolutionary Reinforcement Learning: A Survey
• General Purpose Artificial Intelligence Systems (GPAIS): Properties, Definition, Taxonomy, Societal Implications and Responsible Governance
• Sim2real for Autonomous Vehicle Control using Executable Digital Twin
• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• On the Importance of Reward Design in Reinforcement Learning-based Dynamic Algorithm Configuration: A Case Study on OneMax with (1+($\lambda$,$\lambda$))-GA