Meta-Gradient Reinforcement Learning with an Objective Discovered Online

Abstract

Deep reinforcement learning includes a broad family of algorithms that parameterise an internal representation, such as a value function or policy, by a deep neural network. Each algorithm optimises its parameters with respect to an objective, such as Q-learning or policy gradient, that defines its semantics. In this work, we propose an algorithm based on meta-gradient descent that discovers its own objective, flexibly parameterised by a deep neural network, solely from interactive experience with its environment. Over time, this allows the agent to learn how to learn increasingly effectively. Furthermore, because the objective is discovered online, it can adapt to changes over time. We demonstrate that the algorithm discovers how to address several important issues in RL, such as bootstrapping, non-stationarity, and off-policy learning. On the Atari Learning Environment, the meta-gradient algorithm adapts over time to learn with greater efficiency, eventually outperforming the median score of a strong actor-critic baseline.

References in corpus (13)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Recurrent Models of Visual Attention
• On First-Order Meta-Learning Algorithms
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Learning to reinforcement learn
• Reinforcement Learning with Unsupervised Auxiliary Tasks
• Meta-Gradient Reinforcement Learning
• Off-Policy Actor-Critic
• Discovery of Useful Questions as Auxiliary Tasks
• A Self-Tuning Actor-Critic Algorithm
• On Learning Intrinsic Rewards for Policy Gradient Methods
• Beyond Exponentially Discounted Sum: Automatic Learning of Return Function

Cited by in corpus (12)

• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Discovering Reinforcement Learning Algorithms
• Podracer architectures for scalable Reinforcement Learning
• Offline Meta-Reinforcement Learning with Online Self-Supervision
• Forethought and Hindsight in Credit Assignment
• Training Learned Optimizers with Randomly Initialized Learned Optimizers
• Measuring Progress in Deep Reinforcement Learning Sample Efficiency
• Unifying Gradient Estimators for Meta-Reinforcement Learning via Off-Policy Evaluation
• Learning Pessimism for Robust and Efficient Off-Policy Reinforcement Learning
• Diagnosing and exploiting the computational demands of videos games for deep reinforcement learning
• Neural Auto-Curricula
• One Step at a Time: Pros and Cons of Multi-Step Meta-Gradient Reinforcement Learning