Meta-learning of Sequential Strategies

Abstract

In this report we review memory-based meta-learning as a tool for building sample-efficient strategies that learn from past experience to adapt to any task within a target class. Our goal is to equip the reader with the conceptual foundations of this tool for building new, scalable agents that operate on broad domains. To do so, we present basic algorithmic templates for building near-optimal predictors and reinforcement learners which behave as if they had a probabilistic model that allowed them to efficiently exploit task structure. Furthermore, we recast memory-based meta-learning within a Bayesian framework, showing that the meta-learned strategies are near-optimal because they amortize Bayes-filtered data, where the adaptation is implemented in the memory dynamics as a state-machine of sufficient statistics. Essentially, memory-based meta-learning translates the hard problem of probabilistic sequential inference into a regression problem.

DeepMind Technical Report (15 pages, 6 figures). Version V1.1

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• Prototypical Networks for Few-shot Learning
• Playing Atari with Deep Reinforcement Learning
• A Simple Neural Attentive Meta-Learner
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Learning to reinforcement learn
• Neural Programmer-Interpreters
• Recasting Gradient-Based Meta-Learning as Hierarchical Bayes
• Learning to Optimize
• Sample Efficient Adaptive Text-to-Speech
• Causal Reasoning from Meta-reinforcement Learning
• Learning to Perform Physics Experiments via Deep Reinforcement Learning
• Deep Amortized Inference for Probabilistic Programs
• Posterior Sampling for Reinforcement Learning Without Episodes
• Modeling Friends and Foes

Cited by in corpus (20)

• VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning
• MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale
• Meta-trained agents implement Bayes-optimal agents
• BADGER: Learning to (Learn [Learning Algorithms] through Multi-Agent Communication)
• Recurrent Neural Processes
• Offline Meta Learning of Exploration
• The Synthesizability of Molecules Proposed by Generative Models
• Empirical Bayes Regret Minimization
• Embedding Adaptation is Still Needed for Few-Shot Learning
• Differentiable Bandit Exploration
• Meta-Learning Bandit Policies by Gradient Ascent
• Meta-Thompson Sampling
• Exploration in Approximate Hyper-State Space for Meta Reinforcement Learning
• Hierarchical Bayesian Bandits
• Hierarchical Expert Networks for Meta-Learning
• Unifying Gradient Estimators for Meta-Reinforcement Learning via Off-Policy Evaluation
• What is Going on Inside Recurrent Meta Reinforcement Learning Agents?
• Sparse Meta Networks for Sequential Adaptation and its Application to Adaptive Language Modelling
• Metalearning Linear Bandits by Prior Update
• Model-Free Risk-Sensitive Reinforcement Learning