Orthogonal Convolutional Neural Networks

Abstract

Deep convolutional neural networks are hindered by training instability and feature redundancy towards further performance improvement. A promising solution is to impose orthogonality on convolutional filters. We develop an efficient approach to impose filter orthogonality on a convolutional layer based on the doubly block-Toeplitz matrix representation of the convolutional kernel instead of using the common kernel orthogonality approach, which we show is only necessary but not sufficient for ensuring orthogonal convolutions. Our proposed orthogonal convolution requires no additional parameters and little computational overhead. This method consistently outperforms the kernel orthogonality alternative on a wide range of tasks such as image classification and inpainting under supervised, semi-supervised and unsupervised settings. Further, it learns more diverse and expressive features with better training stability, robustness, and generalization. Our code is publicly available at https://github.com/samaonline/Orthogonal-Convolutional-Neural-Networks.

arXiv commentTo appear in CVPR 2020, project page: http://pwang.pw/ocnn.html

References in corpus (18)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• On the difficulty of training Recurrent Neural Networks
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Spectral Normalization for Generative Adversarial Networks
• Conditional Image Generation with PixelCNN Decoders
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• On orthogonality and learning recurrent networks with long term dependencies
• Simple Black-box Adversarial Attacks
• The Singular Values of Convolutional Layers
• Full-Capacity Unitary Recurrent Neural Networks
• Implicit Generation and Generalization in Energy-Based Models
• Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks
• Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group
• Superposition of many models into one
• Generalized BackPropagation, Étude De Cas: Orthogonality
• Optimization on Submanifolds of Convolution Kernels in CNNs
• DizzyRNN: Reparameterizing Recurrent Neural Networks for Norm-Preserving Backpropagation

Cited by in corpus (4)

• Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training
• Asymptotic Singular Value Distribution of Linear Convolutional Layers
• Regularization for convolutional kernel tensors to avoid unstable gradient problem in convolutional neural networks
• Stochastic Flows and Geometric Optimization on the Orthogonal Group