Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks

Abstract

Orthogonal matrix has shown advantages in training Recurrent Neural Networks (RNNs), but such matrix is limited to be square for the hidden-to-hidden transformation in RNNs. In this paper, we generalize such square orthogonal matrix to orthogonal rectangular matrix and formulating this problem in feed-forward Neural Networks (FNNs) as Optimization over Multiple Dependent Stiefel Manifolds (OMDSM). We show that the rectangular orthogonal matrix can stabilize the distribution of network activations and regularize FNNs. We also propose a novel orthogonal weight normalization method to solve OMDSM. Particularly, it constructs orthogonal transformation over proxy parameters to ensure the weight matrix is orthogonal and back-propagates gradient information through the transformation during training. To guarantee stability, we minimize the distortions between proxy parameters and canonical weights over all tractable orthogonal transformations. In addition, we design an orthogonal linear module (OLM) to learn orthogonal filter banks in practice, which can be used as an alternative to standard linear module. Extensive experiments demonstrate that by simply substituting OLM for standard linear module without revising any experimental protocols, our method largely improves the performance of the state-of-the-art networks, including Inception and residual networks on CIFAR and ImageNet datasets. In particular, we have reduced the test error of wide residual network on CIFAR-100 from 20.04% to 18.61% with such simple substitution. Our code is available online for result reproduction.

20 pages, Accepted by AAAI 2018

Cited by in corpus (39)

• R-Drop: Regularized Dropout for Neural Networks
• Concept Whitening for Interpretable Image Recognition
• ABD-Net: Attentive but Diverse Person Re-Identification
• OSLNet: Deep Small-Sample Classification with an Orthogonal Softmax Layer
• Convolutional Proximal Neural Networks and Plug-and-Play Algorithms
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application
• Parseval Proximal Neural Networks
• Continual Learning in Low-rank Orthogonal Subspaces
• Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transform
• Deep Reinforcement Learning with Robust and Smooth Policy
• Iterative Normalization: Beyond Standardization towards Efficient Whitening
• Orthogonal Relation Transforms with Graph Context Modeling for Knowledge Graph Embedding
• Product of Orthogonal Spheres Parameterization for Disentangled Representation Learning
• Orthogonal Convolutional Neural Networks
• Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training
• Constraint-Based Regularization of Neural Networks
• Deep Learning for Inverse Problems: Bounds and Regularizers
• Improving Molecular Graph Neural Network Explainability with Orthonormalization and Induced Sparsity
• Decentralized Riemannian Gradient Descent on the Stiefel Manifold
• Orthogonal Over-Parameterized Training
• OOGAN: Disentangling GAN with One-Hot Sampling and Orthogonal Regularization
• Self-Orthogonality Module: A Network Architecture Plug-in for Learning Orthogonal Filters
• Regularizing Neural Networks via Minimizing Hyperspherical Energy
• Tractable Density Estimation on Learned Manifolds with Conformal Embedding Flows
• Controllable Orthogonalization in Training DNNs
• Orthogonal Wasserstein GANs
• Deep Asymmetric Networks with a Set of Node-wise Variant Activation Functions
• Preprint: Norm Loss: An efficient yet effective regularization method for deep neural networks
• Pruning Ternary Quantization
• Deep Unitary Convolutional Neural Networks
• From Hard to Soft: Understanding Deep Network Nonlinearities via Vector Quantization and Statistical Inference
• CWY Parametrization: a Solution for Parallelized Optimization of Orthogonal and Stiefel Matrices
• Learning with Hyperspherical Uniformity
• Parallelized Computation and Backpropagation Under Angle-Parametrized Orthogonal Matrices
• Push for Center Learning via Orthogonalization and Subspace Masking for Person Re-Identification
• Data-driven Regularization via Racecar Training for Generalizing Neural Networks
• Hamiltonian Monte-Carlo for Orthogonal Matrices
• Instance-Level Meta Normalization
• Stochastic Flows and Geometric Optimization on the Orthogonal Group