Is the Meta-Learning Idea Able to Improve the Generalization of Deep Neural Networks on the Standard Supervised Learning?

Abstract

Substantial efforts have been made on improving the generalization abilities of deep neural networks (DNNs) in order to obtain better performances without introducing more parameters. On the other hand, meta-learning approaches exhibit powerful generalization on new tasks in few-shot learning. Intuitively, few-shot learning is more challenging than the standard supervised learning as each target class only has a very few or no training samples. The natural question that arises is whether the meta-learning idea can be used for improving the generalization of DNNs on the standard supervised learning. In this paper, we propose a novel meta-learning based training procedure (MLTP) for DNNs and demonstrate that the meta-learning idea can indeed improve the generalization abilities of DNNs. MLTP simulates the meta-training process by considering a batch of training samples as a task. The key idea is that the gradient descent step for improving the current task performance should also improve a new task performance, which is ignored by the current standard procedure for training neural networks. MLTP also benefits from all the existing training techniques such as dropout, weight decay, and batch normalization. We evaluate MLTP by training a variety of small and large neural networks on three benchmark datasets, i.e., CIFAR-10, CIFAR-100, and Tiny ImageNet. The experimental results show a consistently improved generalization performance on all the DNNs with different sizes, which verifies the promise of MLTP and demonstrates that the meta-learning idea is indeed able to improve the generalization of DNNs on the standard supervised learning.

References in corpus (22)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• FitNets: Hints for Thin Deep Nets
• Do Deep Nets Really Need to be Deep?
• Understanding deep learning requires rethinking generalization
• Opening the Black Box of Deep Neural Networks via Information
• On First-Order Meta-Learning Algorithms
• Sensitivity and Generalization in Neural Networks: an Empirical Study
• Robust Large Margin Deep Neural Networks
• Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning
• Recasting Gradient-Based Meta-Learning as Hierarchical Bayes
• Bayesian Model-Agnostic Meta-Learning
• Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
• Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithm
• How to train your MAML
• Online Learning Rate Adaptation with Hypergradient Descent
• Alpha MAML: Adaptive Model-Agnostic Meta-Learning
• Training behavior of deep neural network in frequency domain
• Understanding training and generalization in deep learning by Fourier analysis
• Auto-Meta: Automated Gradient Based Meta Learner Search