Controllable Orthogonalization in Training DNNs
arXiv:2004.00917
Abstract
Orthogonality is widely used for training deep neural networks (DNNs) due to its ability to maintain all singular values of the Jacobian close to 1 and reduce redundancy in representation. This paper proposes a computationally efficient and numerically stable orthogonalization method using Newton's iteration (ONI), to learn a layer-wise orthogonal weight matrix in DNNs. ONI works by iteratively stretching the singular values of a weight matrix towards 1. This property enables it to control the orthogonality of a weight matrix by its number of iterations. We show that our method improves the performance of image classification networks by effectively controlling the orthogonality to provide an optimal tradeoff between optimization benefits and representational capacity reduction. We also show that ONI stabilizes the training of generative adversarial networks (GANs) by maintaining the Lipschitz continuity of a network, similar to spectral normalization (SN), and further outperforms SN by providing controllable orthogonality.
Accepted to CVPR 2020. The Code is available at https://github.com/huangleiBuaa/ONI
References in corpus (20)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- On the difficulty of training Recurrent Neural Networks
- Large Scale GAN Training for High Fidelity Natural Image Synthesis
- Spectral Normalization for Generative Adversarial Networks
- Self-Attention Generative Adversarial Networks
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- cuDNN: Efficient Primitives for Deep Learning
- Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
- Bayesian Uncertainty Estimation for Batch Normalized Deep Networks
- On orthogonality and learning recurrent networks with long term dependencies
- Full-Capacity Unitary Recurrent Neural Networks
- Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks
- CapProNet: Deep Feature Learning via Orthogonal Projections onto Capsule Subspaces
- Training Deep Networks with Structured Layers by Matrix Backpropagation
- Generalized BackPropagation, Étude De Cas: Orthogonality
- Optimization on Submanifolds of Convolution Kernels in CNNs
- DizzyRNN: Reparameterizing Recurrent Neural Networks for Norm-Preserving Backpropagation
- Deep Learning for Inverse Problems: Bounds and Regularizers