Meta-Gradient Reinforcement Learning

Abstract

The goal of reinforcement learning algorithms is to estimate and/or optimise the value function. However, unlike supervised learning, no teacher or oracle is available to provide the true value function. Instead, the majority of reinforcement learning algorithms estimate and/or optimise a proxy for the value function. This proxy is typically based on a sampled and bootstrapped approximation to the true value function, known as a return. The particular choice of return is one of the chief components determining the nature of the algorithm: the rate at which future rewards are discounted; when and how values should be bootstrapped; or even the nature of the rewards themselves. It is well-known that these decisions are crucial to the overall success of RL algorithms. We discuss a gradient-based meta-learning algorithm that is able to adapt the nature of the return, online, whilst interacting and learning from the environment. When applied to 57 games on the Atari 2600 environment over 200 million frames, our algorithm achieved a new state-of-the-art performance.

Cited by in corpus (49)

• SNAS: Stochastic Neural Architecture Search
• Agent57: Outperforming the Atari Human Benchmark
• ETA Prediction with Graph Neural Networks in Google Maps
• Meta-learning PINN loss functions
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Learning to Generalize from Sparse and Underspecified Rewards
• Discovering Reinforcement Learning Algorithms
• learn2learn: A Library for Meta-Learning Research
• Meta-Gradient Reinforcement Learning with an Objective Discovered Online
• Hyp-RL : Hyperparameter Optimization by Reinforcement Learning
• A Self-Tuning Actor-Critic Algorithm
• On Inductive Biases in Deep Reinforcement Learning
• Online Meta-Critic Learning for Off-Policy Actor-Critic Methods
• Learning to Incentivize Other Learning Agents
• When MAML Can Adapt Fast and How to Assist When It Cannot
• Meta-learning curiosity algorithms
• Reinforcement Learning and Adaptive Sampling for Optimized DNN Compilation
• Fast Efficient Hyperparameter Tuning for Policy Gradients
• Muesli: Combining Improvements in Policy Optimization
• Few-shot Quality-Diversity Optimization
• Fast Adaptation via Policy-Dynamics Value Functions
• Offline Meta-Reinforcement Learning with Online Self-Supervision
• Return-based Scaling: Yet Another Normalisation Trick for Deep RL
• Learning Feature Relevance Through Step Size Adaptation in Temporal-Difference Learning
• Distributed Multi-agent Meta Learning for Trajectory Design in Wireless Drone Networks
• A nonlinear hidden layer enables actor-critic agents to learn multiple paired association navigation
• Finding online neural update rules by learning to remember
• Online Hyper-parameter Tuning in Off-policy Learning via Evolutionary Strategies
• Online Meta-Learning for Multi-Source and Semi-Supervised Domain Adaptation
• Mean-Variance Policy Iteration for Risk-Averse Reinforcement Learning
• What Can Learned Intrinsic Rewards Capture?
• Off-Policy Actor-Critic with Shared Experience Replay
• Reward Tweaking: Maximizing the Total Reward While Planning for Short Horizons
• Learning Long-Term Reward Redistribution via Randomized Return Decomposition
• Contextualizing Enhances Gradient Based Meta Learning
• A Primal-Dual Subgradient Approachfor Fair Meta Learning
• Beyond Exponentially Discounted Sum: Automatic Learning of Return Function
• Hindsight Foresight Relabeling for Meta-Reinforcement Learning
• What is Going on Inside Recurrent Meta Reinforcement Learning Agents?
• MDP Playground: An Analysis and Debug Testbed for Reinforcement Learning
• Meta Matrix Factorization for Federated Rating Predictions
• On the Practical Consistency of Meta-Reinforcement Learning Algorithms
• META-Learning Eligibility Traces for More Sample Efficient Temporal Difference Learning
• A nearly Blackwell-optimal policy gradient method
• Towards Practical Credit Assignment for Deep Reinforcement Learning
• Learning to Reweight Imaginary Transitions for Model-Based Reinforcement Learning
• HMRL: Hyper-Meta Learning for Sparse Reward Reinforcement Learning Problem
• Diagnosing and exploiting the computational demands of videos games for deep reinforcement learning
• Bootstrapped Meta-Learning