Meta-Learning without Memorization

Abstract

The ability to learn new concepts with small amounts of data is a critical aspect of intelligence that has proven challenging for deep learning methods. Meta-learning has emerged as a promising technique for leveraging data from previous tasks to enable efficient learning of new tasks. However, most meta-learning algorithms implicitly require that the meta-training tasks be mutually-exclusive, such that no single model can solve all of the tasks at once. For example, when creating tasks for few-shot image classification, prior work uses a per-task random assignment of image classes to N-way classification labels. If this is not done, the meta-learner can ignore the task training data and learn a single model that performs all of the meta-training tasks zero-shot, but does not adapt effectively to new image classes. This requirement means that the user must take great care in designing the tasks, for example by shuffling labels or removing task identifying information from the inputs. In some domains, this makes meta-learning entirely inapplicable. In this paper, we address this challenge by designing a meta-regularization objective using information theory that places precedence on data-driven adaptation. This causes the meta-learner to decide what must be learned from the task training data and what should be inferred from the task testing input. By doing so, our algorithm can successfully use data from non-mutually-exclusive tasks to efficiently adapt to novel tasks. We demonstrate its applicability to both contextual and gradient-based meta-learning algorithms, and apply it in practical settings where applying standard meta-learning has been difficult. Our approach substantially outperforms standard meta-learning algorithms in these settings.

arXiv commentICLR 2020

References in corpus (14)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• The information bottleneck method
• Theoretical Models of Learning to Learn
• Weight Uncertainty in Neural Networks
• Deep Variational Information Bottleneck
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
• Fast Context Adaptation via Meta-Learning
• Recasting Gradient-Based Meta-Learning as Hierarchical Bayes
• Bayesian Model-Agnostic Meta-Learning
• Towards a Neural Statistician
• A PAC-Bayesian bound for Lifelong Learning
• Meta-Learning by Adjusting Priors Based on Extended PAC-Bayes Theory
• Conditional Neural Processes
• Towards Understanding Generalization in Gradient-Based Meta-Learning

Cited by in corpus (14)

• Incremental Object Detection via Meta-Learning
• From Learning to Meta-Learning: Reduced Training Overhead and Complexity for Communication Systems
• End-to-End Fast Training of Communication Links Without a Channel Model via Online Meta-Learning
• Invariant Causal Prediction for Block MDPs
• Data Augmentation for Meta-Learning
• Bayesian Active Meta-Learning for Few Pilot Demodulation and Equalization
• Towards Cross-Granularity Few-Shot Learning: Coarse-to-Fine Pseudo-Labeling with Visual-Semantic Meta-Embedding
• Is Support Set Diversity Necessary for Meta-Learning?
• Non-Gaussian Gaussian Processes for Few-Shot Regression
• Uniform Priors for Data-Efficient Transfer
• PACIA: Parameter-Efficient Adapter for Few-Shot Molecular Property Prediction
• Variable-Shot Adaptation for Online Meta-Learning
• A Few Shot Adaptation of Visual Navigation Skills to New Observations using Meta-Learning
• A Markov Decision Process Approach to Active Meta Learning