Minnorm training: an algorithm for training over-parameterized deep neural networks

Abstract

In this work, we propose a new training method for finding minimum weight norm solutions in over-parameterized neural networks (NNs). This method seeks to improve training speed and generalization performance by framing NN training as a constrained optimization problem wherein the sum of the norm of the weights in each layer of the network is minimized, under the constraint of exactly fitting training data. It draws inspiration from support vector machines (SVMs), which are able to generalize well, despite often having an infinite number of free parameters in their primal form, and from recent theoretical generalization bounds on NNs which suggest that lower norm solutions generalize better. To solve this constrained optimization problem, our method employs Lagrange multipliers that act as integrators of error over training and identify `support vector'-like examples. The method can be implemented as a wrapper around gradient based methods and uses standard back-propagation of gradients from the NN for both regression and classification versions of the algorithm. We provide theoretical justifications for the effectiveness of this algorithm in comparison to early stopping and $L_2$-regularization using simple, analytically tractable settings. In particular, we show faster convergence to the max-margin hyperplane in a shallow network (compared to vanilla gradient descent); faster convergence to the minimum-norm solution in a linear chain (compared to $L_2$-regularization); and initialization-independent generalization performance in a deep linear network. Finally, using the MNIST dataset, we demonstrate that this algorithm can boost test accuracy and identify difficult examples in real-world datasets.

References in corpus (9)

• An Analysis of Deep Neural Network Models for Practical Applications
• Deep Learning using Linear Support Vector Machines
• A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• Spectrally-normalized margin bounds for neural networks
• To understand deep learning we need to understand kernel learning
• Stronger generalization bounds for deep nets via a compression approach
• SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Data
• Convergence of Gradient Descent on Separable Data

Cited by in corpus (6)

• Scaling description of generalization with number of parameters in deep learning
• Learning by Turning: Neural Architecture Aware Optimisation
• Perspective: A Phase Diagram for Deep Learning unifying Jamming, Feature Learning and Lazy Training
• On Generalization of Adaptive Methods for Over-parameterized Linear Regression
• Logit Attenuating Weight Normalization
• How isotropic kernels perform on simple invariants