Discovering Reinforcement Learning Algorithms

Abstract

Reinforcement learning (RL) algorithms update an agent's parameters according to one of several possible rules, discovered manually through years of research. Automating the discovery of update rules from data could lead to more efficient algorithms, or algorithms that are better adapted to specific environments. Although there have been prior attempts at addressing this significant scientific challenge, it remains an open question whether it is feasible to discover alternatives to fundamental concepts of RL such as value functions and temporal-difference learning. This paper introduces a new meta-learning approach that discovers an entire update rule which includes both 'what to predict' (e.g. value functions) and 'how to learn from it' (e.g. bootstrapping) by interacting with a set of environments. The output of this method is an RL algorithm that we call Learned Policy Gradient (LPG). Empirical results show that our method discovers its own alternative to the concept of value functions. Furthermore it discovers a bootstrapping mechanism to maintain and use its predictions. Surprisingly, when trained solely on toy environments, LPG generalises effectively to complex Atari games and achieves non-trivial performance. This shows the potential to discover general RL algorithms from data.

References in corpus (21)

• Continuous control with deep reinforcement learning
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• On First-Order Meta-Learning Algorithms
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Learning to reinforcement learn
• Meta-Gradient Reinforcement Learning
• Distributional Reinforcement Learning with Quantile Regression
• Evolved Policy Gradients
• Improving Generalization in Meta Reinforcement Learning using Learned Objectives
• AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence
• Discovery of Useful Questions as Auxiliary Tasks
• Meta-Gradient Reinforcement Learning with an Objective Discovered Online
• Backpropamine: training self-modifying neural networks with differentiable neuromodulated plasticity
• Behaviour Suite for Reinforcement Learning
• A Self-Tuning Actor-Critic Algorithm
• BADGER: Learning to (Learn [Learning Algorithms] through Multi-Agent Communication)
• Meta-learning curiosity algorithms
• Finding online neural update rules by learning to remember
• Beyond Exponentially Discounted Sum: Automatic Learning of Return Function

Cited by in corpus (14)

• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• Reinforcement Learning with Dual-Observation for General Video Game Playing
• Few-shot Quality-Diversity Optimization
• Podracer architectures for scalable Reinforcement Learning
• Meta Learning Backpropagation And Improving It
• Forethought and Hindsight in Credit Assignment
• Learning Pessimism for Robust and Efficient Off-Policy Reinforcement Learning
• Unifying Gradient Estimators for Meta-Reinforcement Learning via Off-Policy Evaluation
• Policy Gradient RL Algorithms as Directed Acyclic Graphs
• One Step at a Time: Pros and Cons of Multi-Step Meta-Gradient Reinforcement Learning
• Neural Auto-Curricula
• Searching Search Spaces: Meta-evolving a Geometric Encoding for Neural Networks
• Bootstrapped Meta-Learning