Orthogonal Deep Neural Networks
arXiv:1905.05929
Abstract
In this paper, we introduce the algorithms of Orthogonal Deep Neural Networks (OrthDNNs) to connect with recent interest of spectrally regularized deep learning methods. OrthDNNs are theoretically motivated by generalization analysis of modern DNNs, with the aim to find solution properties of network weights that guarantee better generalization. To this end, we first prove that DNNs are of local isometry on data distributions of practical interest; by using a new covering of the sample space and introducing the local isometry property of DNNs into generalization analysis, we establish a new generalization error bound that is both scale- and range-sensitive to singular value spectrum of each of networks' weight matrices. We prove that the optimal bound w.r.t. the degree of isometry is attained when each weight matrix has a spectrum of equal singular values, among which orthogonal weight matrix or a non-square one with orthonormal rows or columns is the most straightforward choice, suggesting the algorithms of OrthDNNs. We present both algorithms of strict and approximate OrthDNNs, and for the later ones we propose a simple yet effective algorithm called Singular Value Bounding (SVB), which performs as well as strict OrthDNNs, but at a much lower computational cost. We also propose Bounded Batch Normalization (BBN) to make compatible use of batch normalization with OrthDNNs. We conduct extensive comparative studies by using modern architectures on benchmark image classification. Experiments show the efficacy of OrthDNNs.
To Appear in IEEE Transactions on Pattern Analysis and Machine Intelligence
References in corpus (17)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Improving neural networks by preventing co-adaptation of feature detectors
- Wide Residual Networks
- On the Number of Linear Regions of Deep Neural Networks
- Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
- Dropout Training as Adaptive Regularization
- Robust Large Margin Deep Neural Networks
- Unitary Evolution Recurrent Neural Networks
- Comparing deep neural networks against humans: object recognition when the signal gets weaker
- Examining the Impact of Blur on Recognition by Convolutional Networks
- Sharp Minima Can Generalize For Deep Nets
- The Singular Values of Convolutional Layers
- Data-Dependent Stability of Stochastic Gradient Descent
- Algorithmic stability and hypothesis complexity
- Optimization on Submanifolds of Convolution Kernels in CNNs
- A Study and Comparison of Human and Deep Learning Recognition Performance Under Visual Distortions
- On the effect of pooling on the geometry of representations