Constraint-Based Regularization of Neural Networks

Abstract

We propose a method for efficiently incorporating constraints into a stochastic gradient Langevin framework for the training of deep neural networks. Constraints allow direct control of the parameter space of the model. Appropriately designed, they reduce the vanishing/exploding gradient problem, control weight magnitudes and stabilize deep neural networks and thus improve the robustness of training algorithms and the generalization capabilities of the trained neural network. We present examples of constrained training methods motivated by orthogonality preservation for weight matrices and explicit weight normalizations. We describe the methods in the overdamped formulation of Langevin dynamics and the underdamped form, in which momenta help to improve sampling efficiency. The methods are explored in test examples in image classification and natural language processing.

T. Vlaar won best student paper award at OPT2020

References in corpus (10)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Improving neural networks by preventing co-adaptation of feature detectors
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• On the difficulty of training Recurrent Neural Networks
• Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks
• Underdamped Langevin MCMC: A non-asymptotic analysis
• Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks
• PyHessian: Neural Networks Through the Lens of the Hessian
• The Implicit and Explicit Regularization Effects of Dropout
• Multiple projection MCMC algorithms on submanifolds

Cited by in corpus (5)

• Solving eigenvalue PDEs of metastable diffusion processes using artificial neural networks
• The shifted ODE method for underdamped Langevin MCMC
• A Unifying and Canonical Description of Measure-Preserving Diffusions
• Informative regularization for a multi-layer perceptron RR Lyrae classifier under data shift
• Non-reversible sampling schemes on submanifolds