Meta-Learning with Network Pruning

Abstract

Meta-learning is a powerful paradigm for few-shot learning. Although with remarkable success witnessed in many applications, the existing optimization based meta-learning models with over-parameterized neural networks have been evidenced to ovetfit on training tasks. To remedy this deficiency, we propose a network pruning based meta-learning approach for overfitting reduction via explicitly controlling the capacity of network. A uniform concentration analysis reveals the benefit of network capacity constraint for reducing generalization gap of the proposed meta-learner. We have implemented our approach on top of Reptile assembled with two network pruning routines: Dense-Sparse-Dense (DSD) and Iterative Hard Thresholding (IHT). Extensive experimental results on benchmark datasets with different over-parameterized deep networks demonstrate that our method not only effectively alleviates meta-overfitting but also in many cases improves the overall generalization performance when applied to few-shot classification tasks.

References in corpus (12)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Prototypical Networks for Few-shot Learning
• Theoretical Models of Learning to Learn
• A Simple Neural Attentive Meta-Learner
• Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
• Meta-Learning for Semi-Supervised Few-Shot Classification
• On First-Order Meta-Learning Algorithms
• Fast Context Adaptation via Meta-Learning
• Learning to Learn: Meta-Critic Networks for Sample Efficient Learning
• Training Skinny Deep Neural Networks with Iterative Hard Thresholding Methods
• Structured Sparsity and Generalization
• Learning-to-Learn Stochastic Gradient Descent with Biased Regularization