Optimization on Submanifolds of Convolution Kernels in CNNs

Abstract

Kernel normalization methods have been employed to improve robustness of optimization methods to reparametrization of convolution kernels, covariate shift, and to accelerate training of Convolutional Neural Networks (CNNs). However, our understanding of theoretical properties of these methods has lagged behind their success in applications. We develop a geometric framework to elucidate underlying mechanisms of a diverse range of kernel normalization methods. Our framework enables us to expound and identify geometry of space of normalized kernels. We analyze and delineate how state-of-the-art kernel normalization methods affect the geometry of search spaces of the stochastic gradient descent (SGD) algorithms in CNNs. Following our theoretical results, we propose a SGD algorithm with assurance of almost sure convergence of the methods to a solution at single minimum of classification loss of CNNs. Experimental results show that the proposed method achieves state-of-the-art performance for major image classification benchmarks with CNNs.

9 pages, 3 figures

References in corpus (9)

• Densely Connected Convolutional Networks
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• Manopt, a Matlab toolbox for optimization on manifolds
• Deep Networks with Stochastic Depth
• Data-dependent Initializations of Convolutional Neural Networks
• Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks
• Natural Neural Networks
• Why Use Sobolev Metrics on the Space of Curves
• Symmetry-invariant optimization in deep networks

Cited by in corpus (17)

• Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group
• ABD-Net: Attentive but Diverse Person Re-Identification
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application
• All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulation
• Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?
• Orthogonal Deep Neural Networks
• Information Geometry of Orthogonal Initializations and Training
• Orthogonal Convolutional Neural Networks
• Controllable Orthogonalization in Training DNNs
• Orthogonal Wasserstein GANs
• Tractable Density Estimation on Learned Manifolds with Conformal Embedding Flows
• Preprint: Norm Loss: An efficient yet effective regularization method for deep neural networks
• Pruning Ternary Quantization
• Optimization on Product Submanifolds of Convolution Kernels
• Data-driven Regularization via Racecar Training for Generalizing Neural Networks
• Parallelized Computation and Backpropagation Under Angle-Parametrized Orthogonal Matrices
• Fine-grained Optimization of Deep Neural Networks