Learning to reinforcement learn

Abstract

In recent years deep reinforcement learning (RL) systems have attained superhuman performance in a number of challenging task domains. However, a major limitation of such applications is their demand for massive amounts of training data. A critical present objective is thus to develop deep RL methods that can adapt rapidly to new tasks. In the present work we introduce a novel approach to this challenge, which we refer to as deep meta-reinforcement learning. Previous work has shown that recurrent networks can support meta-learning in a fully supervised context. We extend this approach to the RL setting. What emerges is a system that is trained using one RL algorithm, but whose recurrent dynamics implement a second, quite separate RL procedure. This second, learned RL algorithm can differ from the original one in arbitrary ways. Importantly, because it is learned, it is configured to exploit structure in the training domain. We unpack these points in a series of seven proof-of-concept experiments, each of which examines a key aspect of deep meta-RL. We consider prospects for extending and scaling up the approach, and also point out some potentially important implications for neuroscience.

17 pages, 7 figures, 1 table

References in corpus (5)

• Neural Architecture Search with Reinforcement Learning
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Learning to learn by gradient descent by gradient descent
• Learning to Optimize
• The Predictron: End-To-End Learning and Planning

Cited by in corpus (218)

• Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
• Searching for Activation Functions
• Solving Rubik's Cube with a Robot Hand
• Unity: A General Platform for Intelligent Agents
• Deep Reinforcement Learning: An Overview
• Advances and Challenges in Conversational Recommender Systems: A Survey
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• Federated Meta-Learning with Fast Convergence and Efficient Communication
• One-Shot Imitation Learning
• Resource Management in Wireless Networks via Multi-Agent Deep Reinforcement Learning
• Meta-Reinforcement Learning of Structured Exploration Strategies
• Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments
• Reinforcement Learning for Angle-Only Intercept Guidance of Maneuvering Targets
• Unsupervised Predictive Memory in a Goal-Directed Agent
• Evolved Policy Gradients
• Adaptive Guidance and Integrated Navigation with Reinforcement Meta-Learning
• Assessing Generalization in Deep Reinforcement Learning
• Learned Optimizers that Scale and Generalize
• Deep Meta-Learning: Learning to Learn in the Concept Space
• Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning
• Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learning
• CEM-RL: Combining evolutionary and gradient-based methods for policy search
• Seeker based Adaptive Guidance via Reinforcement Meta-Learning Applied to Asteroid Close Proximity Operations
• Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithm
• How to train your MAML
• Biologically inspired alternatives to backpropagation through time for learning in recurrent neural nets
• A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation
• Learning to Continually Learn
• Learning to Generalize from Sparse and Underspecified Rewards
• Discovering Reinforcement Learning Algorithms
• Improving Generalization in Meta Reinforcement Learning using Learned Objectives
• Meta reinforcement learning as task inference
• Psychlab: A Psychology Laboratory for Deep Reinforcement Learning Agents
• Six Degree-of-Freedom Body-Fixed Hovering over Unmapped Asteroids via LIDAR Altimetry and Reinforcement Meta-Learning
• Small Sample Learning in Big Data Era
• Neuromorphic Hardware learns to learn
• GeneraLight: Improving Environment Generalization of Traffic Signal Control via Meta Reinforcement Learning
• Unsupervised Meta-Learning for Reinforcement Learning
• Automatic Data Augmentation for Generalization in Deep Reinforcement Learning
• Conditional Neural Processes
• Learning Gradient Descent: Better Generalization and Longer Horizons
• AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence
• Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL
• Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learning
• A neural network walks into a lab: towards using deep nets as models for human behavior
• Compositional generalization through meta sequence-to-sequence learning
• Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions
• Reward Shaping via Meta-Learning
• Quantum Architecture Search with Meta-learning
• A Comprehensive Overview and Survey of Recent Advances in Meta-Learning
• Learning from Few Samples: A Survey
• Meta-Gradient Reinforcement Learning with an Objective Discovered Online
• Few-Shot Goal Inference for Visuomotor Learning and Planning
• MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale
• Meta-learning of Sequential Strategies
• Been There, Done That: Meta-Learning with Episodic Recall
• Meta-learning Pseudo-differential Operators with Deep Neural Networks
• Reinforcement Meta-Learning for Interception of Maneuvering Exoatmospheric Targets with Parasitic Attitude Loop
• AutoLoss: Learning Discrete Schedules for Alternate Optimization
• Parrot: Data-Driven Behavioral Priors for Reinforcement Learning
• Deep Reinforcement Learning amidst Lifelong Non-Stationarity
• Metalearned Neural Memory
• Auto-Meta: Automated Gradient Based Meta Learner Search
• Learning to Explore with Meta-Policy Gradient
• Generalization in Dexterous Manipulation via Geometry-Aware Multi-Task Learning
• Meta-trained agents implement Bayes-optimal agents
• Task-Robust Model-Agnostic Meta-Learning
• Deep Reinforcement Learning in Computer Vision: A Comprehensive Survey
• Reservoirs learn to learn
• Meta-Learning through Hebbian Plasticity in Random Networks
• Guided Feature Transformation (GFT): A Neural Language Grounding Module for Embodied Agents
• Asymmetric self-play for automatic goal discovery in robotic manipulation
• Domain Adaptive Dialog Generation via Meta Learning
• A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions
• The Faults in Our Pi Stars: Security Issues and Open Challenges in Deep Reinforcement Learning
• Deep Learning for Insider Threat Detection: Review, Challenges and Opportunities
• Learning to Learn Variational Semantic Memory
• Unsupervised Learning of Neural Networks to Explain Neural Networks
• Meta-Reinforcement Learning Robust to Distributional Shift via Model Identification and Experience Relabeling
• Meta-Learning for Stochastic Gradient MCMC
• BADGER: Learning to (Learn [Learning Algorithms] through Multi-Agent Communication)
• Differentiable plasticity: training plastic neural networks with backpropagation
• Revisiting Meta-Learning as Supervised Learning
• Sample-Efficient Reinforcement Learning via Counterfactual-Based Data Augmentation
• Watch, Try, Learn: Meta-Learning from Demonstrations and Reward
• Generative One-Shot Learning (GOL): A Semi-Parametric Approach to One-Shot Learning in Autonomous Vision
• Model-based Adversarial Meta-Reinforcement Learning
• Learning to reinforcement learn for Neural Architecture Search
• SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies
• Concurrent Meta Reinforcement Learning
• The effects of negative adaptation in Model-Agnostic Meta-Learning
• Self-Attentional Credit Assignment for Transfer in Reinforcement Learning
• Toward Multimodal Model-Agnostic Meta-Learning
• MHER: Model-based Hindsight Experience Replay
• Meta-learning autoencoders for few-shot prediction
• A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learning
• A Meta-Reinforcement Learning Approach to Process Control
• Dynamic Regret of Policy Optimization in Non-stationary Environments
• Offline Meta-Reinforcement Learning with Advantage Weighting
• M${}^3$RL: Mind-aware Multi-agent Management Reinforcement Learning
• SafeLife 1.0: Exploring Side Effects in Complex Environments
• Multi-task Batch Reinforcement Learning with Metric Learning
• Meta-Learning surrogate models for sequential decision making
• Adaptive Posterior Learning: few-shot learning with a surprise-based memory module
• The Sensory Neuron as a Transformer: Permutation-Invariant Neural Networks for Reinforcement Learning
• Context Meta-Reinforcement Learning via Neuromodulation
• Model-free conventions in multi-agent reinforcement learning with heterogeneous preferences
• Complementary Meta-Reinforcement Learning for Fault-Adaptive Control
• Towards Effective Context for Meta-Reinforcement Learning: an Approach based on Contrastive Learning
• Continuous Meta-Learning without Tasks
• Evolvability ES: Scalable and Direct Optimization of Evolvability
• Generalizing from a few environments in safety-critical reinforcement learning
• Fast Adaptation via Policy-Dynamics Value Functions
• Offline Meta Learning of Exploration
• DMRO:A Deep Meta Reinforcement Learning-based Task Offloading Framework for Edge-Cloud Computing
• Meta Learning Backpropagation And Improving It
• Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrifices
• HALMA: Humanlike Abstraction Learning Meets Affordance in Rapid Problem Solving
• Learning a Prior over Intent via Meta-Inverse Reinforcement Learning
• Evolving Inborn Knowledge For Fast Adaptation in Dynamic POMDP Problems
• Structure Learning in Motor Control:A Deep Reinforcement Learning Model
• Curriculum in Gradient-Based Meta-Reinforcement Learning
• Guiding Policies with Language via Meta-Learning
• Shaking the foundations: delusions in sequence models for interaction and control
• Meta Dialogue Policy Learning
• Learning to Learn with Feedback and Local Plasticity
• A Decentralized Policy Gradient Approach to Multi-task Reinforcement Learning
• Causal Analysis of Agent Behavior for AI Safety
• Finding online neural update rules by learning to remember
• Guarantees for Tuning the Step Size using a Learning-to-Learn Approach
• Meta Automatic Curriculum Learning
• Generalized Decision Transformer for Offline Hindsight Information Matching
• Variational Autoencoders for Opponent Modeling in Multi-Agent Systems
• StarCraft Micromanagement with Reinforcement Learning and Curriculum Transfer Learning
• Building Machines that Learn and Think for Themselves: Commentary on Lake et al., Behavioral and Brain Sciences, 2017
• Meta-Learning of Structured Task Distributions in Humans and Machines
• Genetic algorithms with DNN-based trainable crossover as an example of partial specialization of general search
• What Can Learned Intrinsic Rewards Capture?
• A Survey of Exploration Methods in Reinforcement Learning
• A Model-based Approach for Sample-efficient Multi-task Reinforcement Learning
• Procedural Generalization by Planning with Self-Supervised World Models
• Meta-Reinforcement Learning in Broad and Non-Parametric Environments
• Learning Navigation Behaviors End-to-End with AutoRL
• Agent Modelling under Partial Observability for Deep Reinforcement Learning
• Meta-Model-Based Meta-Policy Optimization
• The Adversarial Resilience Learning Architecture for AI-based Modelling, Exploration, and Operation of Complex Cyber-Physical Systems
• Meta-Learning to Cluster
• On Machine Learning and Structure for Mobile Robots
• Self-organization of action hierarchy and compositionality by reinforcement learning with recurrent neural networks
• Learning State Abstractions for Transfer in Continuous Control
• Intelligence, physics and information -- the tradeoff between accuracy and simplicity in machine learning
• Bottom-Up Meta-Policy Search
• EVO-RL: Evolutionary-Driven Reinforcement Learning
• REALab: An Embedded Perspective on Tampering
• Adaptive Adversarial Training for Meta Reinforcement Learning
• MELD: Meta-Reinforcement Learning from Images via Latent State Models
• Meta Learning MPC using Finite-Dimensional Gaussian Process Approximations
• Meta Reinforcement Learning with Autonomous Inference of Subtask Dependencies
• The Information Geometry of Unsupervised Reinforcement Learning
• On the Convergence Theory of Debiased Model-Agnostic Meta-Reinforcement Learning
• Group Equivariant Deep Reinforcement Learning
• A Brief Look at Generalization in Visual Meta-Reinforcement Learning
• Testing the Genomic Bottleneck Hypothesis in Hebbian Meta-Learning
• On the Reliability and Generalizability of Brain-inspired Reinforcement Learning Algorithms
• OCEAN: Online Task Inference for Compositional Tasks with Context Adaptation
• Model-Agnostic Meta-Learning using Runge-Kutta Methods
• Task-Agnostic Online Reinforcement Learning with an Infinite Mixture of Gaussian Processes
• Provably Improved Context-Based Offline Meta-RL with Attention and Contrastive Learning
• Continual and Multi-task Reinforcement Learning With Shared Episodic Memory
• Hindsight Foresight Relabeling for Meta-Reinforcement Learning
• Sparse Attention Guided Dynamic Value Estimation for Single-Task Multi-Scene Reinforcement Learning
• Exploration by Maximizing Rényi Entropy for Reward-Free RL Framework
• Improving Context-Based Meta-Reinforcement Learning with Self-Supervised Trajectory Contrastive Learning
• Meta-Learning with Hessian-Free Approach in Deep Neural Nets Training
• Quick Learner Automated Vehicle Adapting its Roadmanship to Varying Traffic Cultures with Meta Reinforcement Learning
• Variable-Shot Adaptation for Online Meta-Learning
• Learning model-based strategies in simple environments with hierarchical q-networks
• Neural Conditional Gradients
• Local Nonparametric Meta-Learning
• Meta-Reinforcement Learning for Heuristic Planning
• Meta-Adversarial Inverse Reinforcement Learning for Decision-making Tasks
• Learning MDPs from Features: Predict-Then-Optimize for Sequential Decision Problems by Reinforcement Learning
• Zero-shot task adaptation by homoiconic meta-mapping
• Interactive Agent Modeling by Learning to Probe
• Learning Meta Representations for Agents in Multi-Agent Reinforcement Learning
• What is Going on Inside Recurrent Meta Reinforcement Learning Agents?
• Deep Episodic Value Iteration for Model-based Meta-Reinforcement Learning
• A Useful Motif for Flexible Task Learning in an Embodied Two-Dimensional Visual Environment
• General AI Challenge - Round One: Gradual Learning
• Modular Continual Learning in a Unified Visual Environment
• One-Shot Generation of Near-Optimal Topology through Theory-Driven Machine Learning
• HMRL: Hyper-Meta Learning for Sparse Reward Reinforcement Learning Problem
• When Autonomous Systems Meet Accuracy and Transferability through AI: A Survey
• Planning with Brain-inspired AI
• Submodular Meta-Learning
• Recurrent Neural-Linear Posterior Sampling for Nonstationary Contextual Bandits
• A Meta-Learning Control Algorithm with Provable Finite-Time Guarantees
• Meta-Learning Guarantees for Online Receding Horizon Learning Control
• Transfer Reinforcement Learning across Homotopy Classes
• Learning Shared Dynamics with Meta-World Models
• Meta-Learning with Adjoint Methods
• On the Practical Consistency of Meta-Reinforcement Learning Algorithms
• Neural Auto-Curricula
• Adaptable image quality assessment using meta-reinforcement learning of task amenability
• Eden: A Unified Environment Framework for Booming Reinforcement Learning Algorithms
• Hindsight Reward Tweaking via Conditional Deep Reinforcement Learning
• Knowledge is reward: Learning optimal exploration by predictive reward cashing
• Behaviour-conditioned policies for cooperative reinforcement learning tasks
• MetalGAN: a Cluster-based Adaptive Training for Few-Shot Adversarial Colorization
• Model-Free Risk-Sensitive Reinforcement Learning
• L2E: Learning to Exploit Your Opponent
• Generalized Reinforcement Meta Learning for Few-Shot Optimization
• Towards intervention-centric causal reasoning in learning agents
• Meta Learning in the Continuous Time Limit
• MTL2L: A Context Aware Neural Optimiser
• Learning Data-Driven Objectives to Optimize Interactive Systems
• Agent Probing Interaction Policies
• Where Do Human Heuristics Come From?