RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning

Abstract

Deep reinforcement learning (deep RL) has been successful in learning sophisticated behaviors automatically; however, the learning process requires a huge number of trials. In contrast, animals can learn new tasks in just a few trials, benefiting from their prior knowledge about the world. This paper seeks to bridge this gap. Rather than designing a "fast" reinforcement learning algorithm, we propose to represent it as a recurrent neural network (RNN) and learn it from data. In our proposed method, RL${}^2$, the algorithm is encoded in the weights of the RNN, which are learned slowly through a general-purpose ("slow") RL algorithm. The RNN receives all information a typical RL algorithm would receive, including observations, actions, rewards, and termination flags; and it retains its state across episodes in a given Markov Decision Process (MDP). The activations of the RNN store the state of the "fast" RL algorithm on the current (previously unseen) MDP. We evaluate RL${}^2$ experimentally on both small-scale and large-scale problems. On the small-scale side, we train it to solve randomly generated multi-arm bandit problems and finite MDPs. After RL${}^2$ is trained, its performance on new MDPs is close to human-designed algorithms with optimality guarantees. On the large-scale side, we test RL${}^2$ on a vision-based navigation task and show that it scales up to high-dimensional problems.

14 pages. Under review as a conference paper at ICLR 2017

Cited by in corpus (38)

• A Brief Survey of Deep Reinforcement Learning
• Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
• Searching for Activation Functions
• Learning to reinforcement learn
• Neural Interactive Collaborative Filtering
• Generalized Inner Loop Meta-Learning
• Biologically inspired alternatives to backpropagation through time for learning in recurrent neural nets
• Learning to Continually Learn
• Large-Scale Long-Tailed Recognition in an Open World
• VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning
• Improving Generalization in Meta Reinforcement Learning using Learned Objectives
• GeneraLight: Improving Environment Generalization of Traffic Signal Control via Meta Reinforcement Learning
• Deep Learning Theory Review: An Optimal Control and Dynamical Systems Perspective
• Reward Shaping via Meta-Learning
• Learning Exploration Policies for Navigation
• Learning from Few Samples: A Survey
• Meta-Gradient Reinforcement Learning with an Objective Discovered Online
• Meta-learning of Sequential Strategies
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Hemingway: Modeling Distributed Optimization Algorithms
• Revisiting Meta-Learning as Supervised Learning
• Concurrent Meta Reinforcement Learning
• Experience-Embedded Visual Foresight
• Toward Multimodal Model-Agnostic Meta-Learning
• Meta Reinforcement Learning with Task Embedding and Shared Policy
• Generalizing from a few environments in safety-critical reinforcement learning
• Meta Inverse Reinforcement Learning via Maximum Reward Sharing for Human Motion Analysis
• Building Machines that Learn and Think for Themselves: Commentary on Lake et al., Behavioral and Brain Sciences, 2017
• Variational Autoencoders for Opponent Modeling in Multi-Agent Systems
• Generalized Hidden Parameter MDPs Transferable Model-based RL in a Handful of Trials
• A Brief Look at Generalization in Visual Meta-Reinforcement Learning
• Hierarchical Meta Learning
• Continual and Multi-task Reinforcement Learning With Shared Episodic Memory
• OCEAN: Online Task Inference for Compositional Tasks with Context Adaptation
• VIABLE: Fast Adaptation via Backpropagating Learned Loss
• Local Nonparametric Meta-Learning
• Where Do Human Heuristics Come From?
• On the Possibility of Rewarding Structure Learning Agents: Mutual Information on Linguistic Random Sets