Minimum weight norm models do not always generalize well for over-parameterized problems

Abstract

This work is substituted by the paper in arXiv:2011.14066. Stochastic gradient descent is the de facto algorithm for training deep neural networks (DNNs). Despite its popularity, it still requires fine tuning in order to achieve its best performance. This has led to the development of adaptive methods, that claim automatic hyper-parameter optimization. Recently, researchers have studied both algorithmic classes via toy examples: e.g., for over-parameterized linear regression, Wilson et. al. (2017) shows that, while SGD always converges to the minimum-norm solution, adaptive methods show no such inclination, leading to worse generalization capabilities. Our aim is to study this conjecture further. We empirically show that the minimum weight norm is not necessarily the proper gauge of good generalization in simplified scenaria, and different models found by adaptive methods could outperform plain gradient methods. In practical DNN settings, we observe that adaptive methods can outperform SGD, with larger weight norm output models, but without necessarily reducing the amount of tuning required.

arXiv commentThis work is substituted by the paper in arXiv:2011.14066

References in corpus (4)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Don't Decay the Learning Rate, Increase the Batch Size
• Towards Optimal One Pass Large Scale Learning with Averaged Stochastic Gradient Descent
• Variants of RMSProp and Adagrad with Logarithmic Regret Bounds

Cited by in corpus (9)

• Implicit Regularization in Deep Learning May Not Be Explainable by Norms
• Identity Crisis: Memorization and Generalization under Extreme Overparameterization
• Bad Global Minima Exist and SGD Can Reach Them
• Riemannian adaptive stochastic gradient algorithms on matrix manifolds
• Optimizer Benchmarking Needs to Account for Hyperparameter Tuning
• Truth or Backpropaganda? An Empirical Investigation of Deep Learning Theory
• Linear Convergence of Adaptive Stochastic Gradient Descent
• Which Minimizer Does My Neural Network Converge To?
• How much progress have we made in neural network training? A New Evaluation Protocol for Benchmarking Optimizers