Improving Generalization in Meta Reinforcement Learning using Learned Objectives

Abstract

Biological evolution has distilled the experiences of many learners into the general learning algorithms of humans. Our novel meta reinforcement learning algorithm MetaGenRL is inspired by this process. MetaGenRL distills the experiences of many complex agents to meta-learn a low-complexity neural objective function that decides how future individuals will learn. Unlike recent meta-RL algorithms, MetaGenRL can generalize to new environments that are entirely different from those used for meta-training. In some cases, it even outperforms human-engineered RL algorithms. MetaGenRL uses off-policy second-order gradients during meta-training that greatly increase its sample efficiency.

Accepted to ICLR 2020

References in corpus (20)

• Auto-Encoding Variational Bayes
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Playing Atari with Deep Reinforcement Learning
• Trust Region Policy Optimization
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• A Simple Neural Attentive Meta-Learner
• Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Recurrent World Models Facilitate Policy Evolution
• Learning to reinforcement learn
• Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning
• Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research
• Recasting Gradient-Based Meta-Learning as Hierarchical Bayes
• Learning to Optimize
• Learning to Learn: Meta-Critic Networks for Sample Efficient Learning
• Policy Distillation
• Gotta Learn Fast: A New Benchmark for Generalization in RL
• Learning to Optimize Neural Nets
• AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence
• Meta-learning curiosity algorithms

Cited by in corpus (31)

• Meta-learning PINN loss functions
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Discovering Reinforcement Learning Algorithms
• GeneraLight: Improving Environment Generalization of Traffic Signal Control via Meta Reinforcement Learning
• A Comprehensive Overview and Survey of Recent Advances in Meta-Learning
• Materials Discovery with Extreme Properties via Reinforcement Learning-Guided Combinatorial Chemistry
• Evolving Reinforcement Learning Algorithms
• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• BADGER: Learning to (Learn [Learning Algorithms] through Multi-Agent Communication)
• Model-based Adversarial Meta-Reinforcement Learning
• Offline Meta-Reinforcement Learning with Advantage Weighting
• Multi-task Batch Reinforcement Learning with Metric Learning
• One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL
• Meta Learning Backpropagation And Improving It
• Offline Meta-Reinforcement Learning with Online Self-Supervision
• Grounding Artificial Intelligence in the Origins of Human Behavior
• What Can Learned Intrinsic Rewards Capture?
• Procedural Generalization by Planning with Self-Supervised World Models
• Evaluating the progress of Deep Reinforcement Learning in the real world: aligning domain-agnostic and domain-specific research
• REIN-2: Giving Birth to Prepared Reinforcement Learning Agents Using Reinforcement Learning Agents
• Improving Generalization in Meta-RL with Imaginary Tasks from Latent Dynamics Mixture
• A Brief Look at Generalization in Visual Meta-Reinforcement Learning
• Learning Associative Inference Using Fast Weight Memory
• Uniform Priors for Data-Efficient Transfer
• Group Equivariant Deep Reinforcement Learning
• Improved Robustness and Safety for Pre-Adaptation of Meta Reinforcement Learning with Prior Regularization
• Automatically Learning Fallback Strategies with Model-Free Reinforcement Learning in Safety-Critical Driving Scenarios
• Learning Provably Robust Motion Planners Using Funnel Libraries
• Policy Gradient RL Algorithms as Directed Acyclic Graphs
• Neural Auto-Curricula
• One Step at a Time: Pros and Cons of Multi-Step Meta-Gradient Reinforcement Learning