RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning

Abstract

Deep reinforcement learning (deep RL) has been successful in learning sophisticated behaviors automatically; however, the learning process requires a huge number of trials. In contrast, animals can learn new tasks in just a few trials, benefiting from their prior knowledge about the world. This paper seeks to bridge this gap. Rather than designing a "fast" reinforcement learning algorithm, we propose to represent it as a recurrent neural network (RNN) and learn it from data. In our proposed method, RL${}^2$, the algorithm is encoded in the weights of the RNN, which are learned slowly through a general-purpose ("slow") RL algorithm. The RNN receives all information a typical RL algorithm would receive, including observations, actions, rewards, and termination flags; and it retains its state across episodes in a given Markov Decision Process (MDP). The activations of the RNN store the state of the "fast" RL algorithm on the current (previously unseen) MDP. We evaluate RL${}^2$ experimentally on both small-scale and large-scale problems. On the small-scale side, we train it to solve randomly generated multi-arm bandit problems and finite MDPs. After RL${}^2$ is trained, its performance on new MDPs is close to human-designed algorithms with optimality guarantees. On the large-scale side, we test RL${}^2$ on a vision-based navigation task and show that it scales up to high-dimensional problems.

14 pages. Under review as a conference paper at ICLR 2017

References in corpus (4)

• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Asynchronous Methods for Deep Reinforcement Learning
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Benchmarking Deep Reinforcement Learning for Continuous Control

Cited by in corpus (173)

• A Brief Survey of Deep Reinforcement Learning
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
• Searching for Activation Functions
• How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned
• Learning to reinforcement learn
• Advances and Challenges in Conversational Recommender Systems: A Survey
• Federated Meta-Learning with Fast Convergence and Efficient Communication
• Reinforcement Learning with Augmented Data
• One-Shot Imitation Learning
• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• Bayesian Model-Agnostic Meta-Learning
• Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments
• Evolved Policy Gradients
• Neural Interactive Collaborative Filtering
• Assessing Generalization in Deep Reinforcement Learning
• Review: Deep Learning in Electron Microscopy
• Deep Meta-Learning: Learning to Learn in the Concept Space
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithm
• Generalized Inner Loop Meta-Learning
• Efficient Exploration via State Marginal Matching
• Machine learning for industrial sensing and control: A survey and practical perspective
• Biologically inspired alternatives to backpropagation through time for learning in recurrent neural nets
• Learning to Continually Learn
• Learning to Generalize from Sparse and Underspecified Rewards
• Some Considerations on Learning to Explore via Meta-Reinforcement Learning
• VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning
• Large-Scale Long-Tailed Recognition in an Open World
• Discovering Reinforcement Learning Algorithms
• Improving Generalization in Meta Reinforcement Learning using Learned Objectives
• Dynamical learning of dynamics
• Psychlab: A Psychology Laboratory for Deep Reinforcement Learning Agents
• Small Sample Learning in Big Data Era
• Unsupervised Meta-Learning for Reinforcement Learning
• GeneraLight: Improving Environment Generalization of Traffic Signal Control via Meta Reinforcement Learning
• Image quality assessment for machine learning tasks using meta-reinforcement learning
• Learning Scheduling Algorithms for Data Processing Clusters
• Deep reinforcement learning for machine scheduling: Methodology, the state-of-the-art, and future directions
• AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence
• Deep Learning Theory Review: An Optimal Control and Dynamical Systems Perspective
• Meta-Reinforcement Learning for the Tuning of PI Controllers: An Offline Approach
• Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions
• Reward Shaping via Meta-Learning
• Learning Exploration Policies for Navigation
• Learning from Few Samples: A Survey
• Meta-Gradient Reinforcement Learning with an Objective Discovered Online
• Grounded Language Learning Fast and Slow
• Few-Shot Goal Inference for Visuomotor Learning and Planning
• Prior-Knowledge and Attention-based Meta-Learning for Few-Shot Learning
• Meta-learning of Sequential Strategies
• Been There, Done That: Meta-Learning with Episodic Recall
• Modular meta-learning
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Parrot: Data-Driven Behavioral Priors for Reinforcement Learning
• Deep Reinforcement Learning amidst Lifelong Non-Stationarity
• Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learning
• Meta-Learning by the Baldwin Effect
• Learning to Explore with Meta-Policy Gradient
• Generalization in Dexterous Manipulation via Geometry-Aware Multi-Task Learning
• Reservoirs learn to learn
• Robust MAML: Prioritization task buffer with adaptive learning process for model-agnostic meta-learning
• Probabilistic Model-Agnostic Meta-Learning
• Hemingway: Modeling Distributed Optimization Algorithms
• Infusing model predictive control into meta-reinforcement learning for mobile robots in dynamic environments
• Asymmetric self-play for automatic goal discovery in robotic manipulation
• Internet Congestion Control via Deep Reinforcement Learning
• Meta-Reinforcement Learning Robust to Distributional Shift via Model Identification and Experience Relabeling
• Learning to Learn Variational Semantic Memory
• Regularization Matters in Policy Optimization
• Meta-Learning for Stochastic Gradient MCMC
• Deep Curiosity Search: Intra-Life Exploration Can Improve Performance on Challenging Deep Reinforcement Learning Problems
• Sample-Efficient Reinforcement Learning via Counterfactual-Based Data Augmentation
• Revisiting Meta-Learning as Supervised Learning
• Watch, Try, Learn: Meta-Learning from Demonstrations and Reward
• Concurrent Meta Reinforcement Learning
• Experience-Embedded Visual Foresight
• Toward Multimodal Model-Agnostic Meta-Learning
• A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learning
• Meta-learning autoencoders for few-shot prediction
• Learning Fast Adaptation with Meta Strategy Optimization
• Lifelong Policy Gradient Learning of Factored Policies for Faster Training Without Forgetting
• Offline Meta-Reinforcement Learning with Advantage Weighting
• Procedural Content Generation: Better Benchmarks for Transfer Reinforcement Learning
• Dynamic Regret of Policy Optimization in Non-stationary Environments
• Few-shot Quality-Diversity Optimization
• Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers
• Learning and Planning with a Semantic Model
• Context Meta-Reinforcement Learning via Neuromodulation
• Multi-task Batch Reinforcement Learning with Metric Learning
• The Sensory Neuron as a Transformer: Permutation-Invariant Neural Networks for Reinforcement Learning
• One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL
• Modeling and Optimization Trade-off in Meta-learning
• MetaCURE: Meta Reinforcement Learning with Empowerment-Driven Exploration
• Offline Meta Learning of Exploration
• Generalizing from a few environments in safety-critical reinforcement learning
• Meta Reinforcement Learning with Task Embedding and Shared Policy
• Evolving Inborn Knowledge For Fast Adaptation in Dynamic POMDP Problems
• Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrifices
• Meta Learning Backpropagation And Improving It
• Offline Meta-Reinforcement Learning with Online Self-Supervision
• Thief, Beware of What Get You There: Towards Understanding Model Extraction Attack
• Context-Aware Safe Reinforcement Learning for Non-Stationary Environments
• Meta Inverse Reinforcement Learning via Maximum Reward Sharing for Human Motion Analysis
• Shaking the foundations: delusions in sequence models for interaction and control
• Trajectory-wise Multiple Choice Learning for Dynamics Generalization in Reinforcement Learning
• Bayesian Relational Memory for Semantic Visual Navigation
• Policy Gradient Optimization of Thompson Sampling Policies
• Finding online neural update rules by learning to remember
• Genetic algorithms with DNN-based trainable crossover as an example of partial specialization of general search
• Guarantees for Tuning the Step Size using a Learning-to-Learn Approach
• Building Machines that Learn and Think for Themselves: Commentary on Lake et al., Behavioral and Brain Sciences, 2017
• A Survey of Exploration Methods in Reinforcement Learning
• A Model-based Approach for Sample-efficient Multi-task Reinforcement Learning
• Mutual Information Based Knowledge Transfer Under State-Action Dimension Mismatch
• What Can Learned Intrinsic Rewards Capture?
• Meta Learning Black-Box Population-Based Optimizers
• Variational Autoencoders for Opponent Modeling in Multi-Agent Systems
• Conservative Predictions on Noisy Financial Data
• Few-Shot Learning Patterns in Financial Time-Series for Trend-Following Strategies
• MetaFuse: A Pre-trained Fusion Model for Human Pose Estimation
• Adaptive Asynchronous Control Using Meta-learned Neural Ordinary Differential Equations
• Generalized Hidden Parameter MDPs Transferable Model-based RL in a Handful of Trials
• REALab: An Embedded Perspective on Tampering
• Adaptive Adversarial Training for Meta Reinforcement Learning
• REPAINT: Knowledge Transfer in Deep Reinforcement Learning
• Exploration in Approximate Hyper-State Space for Meta Reinforcement Learning
• MELD: Meta-Reinforcement Learning from Images via Latent State Models
• Zero-Shot Compositional Policy Learning via Language Grounding
• Sparse Attention Guided Dynamic Value Estimation for Single-Task Multi-Scene Reinforcement Learning
• Continual and Multi-task Reinforcement Learning With Shared Episodic Memory
• Off-Policy Meta-Reinforcement Learning Based on Feature Embedding Spaces
• Quick Learner Automated Vehicle Adapting its Roadmanship to Varying Traffic Cultures with Meta Reinforcement Learning
• Population-Based Evolution Optimizes a Meta-Learning Objective
• Learning to Learn and Predict: A Meta-Learning Approach for Multi-Label Classification
• A Brief Look at Generalization in Visual Meta-Reinforcement Learning
• Dynamic Value Estimation for Single-Task Multi-Scene Reinforcement Learning
• OCEAN: Online Task Inference for Compositional Tasks with Context Adaptation
• Hindsight Foresight Relabeling for Meta-Reinforcement Learning
• Testing the Genomic Bottleneck Hypothesis in Hebbian Meta-Learning
• VIABLE: Fast Adaptation via Backpropagating Learned Loss
• Improving Reliable Navigation under Uncertainty via Predictions Informed by Non-Local Information
• Hierarchical Meta Learning
• Improved Robustness and Safety for Pre-Adaptation of Meta Reinforcement Learning with Prior Regularization
• Learning Meta Representations for Agents in Multi-Agent Reinforcement Learning
• Variable-Shot Adaptation for Online Meta-Learning
• Local Nonparametric Meta-Learning
• Fighting Copycat Agents in Behavioral Cloning from Observation Histories
• Learning to Model Opponent Learning
• Meta-Reinforcement Learning for Heuristic Planning
• What is Going on Inside Recurrent Meta Reinforcement Learning Agents?
• Learning MDPs from Features: Predict-Then-Optimize for Sequential Decision Problems by Reinforcement Learning
• Adaptive Submodular Meta-Learning
• MetalGAN: a Cluster-based Adaptive Training for Few-Shot Adversarial Colorization
• On Memory Mechanism in Multi-Agent Reinforcement Learning
• Recurrent Neural-Linear Posterior Sampling for Nonstationary Contextual Bandits
• A Meta-Learning Control Algorithm with Provable Finite-Time Guarantees
• Meta-Learning Guarantees for Online Receding Horizon Learning Control
• Generalized Reinforcement Meta Learning for Few-Shot Optimization
• Learning Shared Dynamics with Meta-World Models
• Transfer Reinforcement Learning across Homotopy Classes
• Performance-Weighed Policy Sampling for Meta-Reinforcement Learning
• On the Possibility of Rewarding Structure Learning Agents: Mutual Information on Linguistic Random Sets
• HMRL: Hyper-Meta Learning for Sparse Reward Reinforcement Learning Problem
• Meta-Learning with Adjoint Methods
• Enhanced Scene Specificity with Sparse Dynamic Value Estimation
• Hindsight Reward Tweaking via Conditional Deep Reinforcement Learning
• Where Do Human Heuristics Come From?
• Learning Data-Driven Objectives to Optimize Interactive Systems
• Behaviour-conditioned policies for cooperative reinforcement learning tasks
• Adaptation of Quadruped Robot Locomotion with Meta-Learning
• Adaptable image quality assessment using meta-reinforcement learning of task amenability
• Knowledge is reward: Learning optimal exploration by predictive reward cashing