Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?

Abstract

This paper seeks to answer the question: as the (near-) orthogonality of weights is found to be a favorable property for training deep convolutional neural networks, how can we enforce it in more effective and easy-to-use ways? We develop novel orthogonality regularizations on training deep CNNs, utilizing various advanced analytical tools such as mutual coherence and restricted isometry property. These plug-and-play regularizations can be conveniently incorporated into training almost any CNN without extra hassle. We then benchmark their effects on state-of-the-art models: ResNet, WideResNet, and ResNeXt, on several most popular computer vision datasets: CIFAR-10, CIFAR-100, SVHN and ImageNet. We observe consistent performance gains after applying those proposed regularizations, in terms of both the final accuracies achieved, and faster and more stable convergences. We have made our codes and pre-trained models publicly available: https://github.com/nbansal90/Can-we-Gain-More-from-Orthogonality.

arXiv comment11 pages, 1 figure, 2 tables. Accepted in NIPS 2018

References in corpus (19)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Spectral Normalization for Generative Adversarial Networks
• Wide Residual Networks
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Robust Large Margin Deep Neural Networks
• Spectral Norm Regularization for Improving the Generalizability of Deep Learning
• All you need is a good init
• On orthogonality and learning recurrent networks with long term dependencies
• Regularizing CNNs with Locally Constrained Decorrelations
• Natural Neural Networks
• Generalized BackPropagation, Étude De Cas: Orthogonality
• Optimization on Submanifolds of Convolution Kernels in CNNs
• All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulation
• Efficient Orthogonal Parametrisation of Recurrent Neural Networks Using Householder Reflections
• DizzyRNN: Reparameterizing Recurrent Neural Networks for Norm-Preserving Backpropagation
• Optimized Projections for Compressed Sensing via Direct Mutual Coherence Minimization
• Learning Simple Thresholded Features with Sparse Support Recovery

Cited by in corpus (4)

• Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group
• Classical and Quantum Algorithms for Orthogonal Neural Networks
• SuperFront: From Low-resolution to High-resolution Frontal Face Synthesis
• Uncertainty Surrogates for Deep Learning