Meta-Learning Requires Meta-Augmentation

Abstract

Meta-learning algorithms aim to learn two components: a model that predicts targets for a task, and a base learner that quickly updates that model when given examples from a new task. This additional level of learning can be powerful, but it also creates another potential source for overfitting, since we can now overfit in either the model or the base learner. We describe both of these forms of metalearning overfitting, and demonstrate that they appear experimentally in common meta-learning benchmarks. We then use an information-theoretic framework to discuss meta-augmentation, a way to add randomness that discourages the base learner and model from learning trivial solutions that do not generalize to new tasks. We demonstrate that meta-augmentation produces large complementary benefits to recently proposed meta-regularization techniques.

arXiv comment14 pages, 8 figures. NeurIPS 2020 camera ready. Code at https://github.com/google-research/google-research/tree/master/meta_augmentation

References in corpus (13)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Prototypical Networks for Few-shot Learning
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• Sensitivity and Generalization in Neural Networks: an Empirical Study
• One-Shot Visual Imitation Learning via Meta-Learning
• Reinforcement Learning with Augmented Data
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• Meta-Learning Update Rules for Unsupervised Representation Learning
• Generative Adversarial Residual Pairwise Networks for One Shot Learning
• Unsupervised Meta-Learning For Few-Shot Image Classification
• Assume, Augment and Learn: Unsupervised Few-Shot Meta-Learning via Random Labels and Data Augmentation
• Task Augmentation by Rotating for Meta-Learning
• Towards Understanding Generalization in Gradient-Based Meta-Learning

Cited by in corpus (8)

• BOIL: Towards Representation Change for Few-shot Learning
• Data Augmentation for Meta-Learning
• How to Train Your MAML to Excel in Few-Shot Classification
• Improving Generalization in Meta-learning via Task Augmentation
• Meta-Learning with Fewer Tasks through Task Interpolation
• Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environment
• Vision-Based Hand Gesture Customization from a Single Demonstration
• Reducing and Exploiting Data Augmentation Noise through Meta Reweighting Contrastive Learning for Text Classification