TaskNorm: Rethinking Batch Normalization for Meta-Learning
arXiv:2003.03284
Abstract
Modern meta-learning approaches for image classification rely on increasingly deep networks to achieve state-of-the-art performance, making batch normalization an essential component of meta-learning pipelines. However, the hierarchical nature of the meta-learning setting presents several challenges that can render conventional batch normalization ineffective, giving rise to the need to rethink normalization in this setting. We evaluate a range of approaches to batch normalization for meta-learning scenarios, and develop a novel approach that we call TaskNorm. Experiments on fourteen datasets demonstrate that the choice of batch normalization has a dramatic effect on both classification accuracy and training time for both gradient based and gradient-free meta-learning approaches. Importantly, TaskNorm is found to consistently improve performance. Finally, we provide a set of best practices for normalization that will allow fair comparison of meta-learning algorithms.
References in corpus (12)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Prototypical Networks for Few-shot Learning
- Instance Normalization: The Missing Ingredient for Fast Stylization
- Theoretical Models of Learning to Learn
- Matching Networks for One Shot Learning
- A Simple Neural Attentive Meta-Learner
- Learning to Propagate Labels: Transductive Propagation Network for Few-shot Learning
- On First-Order Meta-Learning Algorithms
- Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
- Meta-Learning with Implicit Gradients
- Recasting Gradient-Based Meta-Learning as Hierarchical Bayes
- Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL