Projection Based Weight Normalization for Deep Neural Networks

Abstract

Optimizing deep neural networks (DNNs) often suffers from the ill-conditioned problem. We observe that the scaling-based weight space symmetry property in rectified nonlinear network will cause this negative effect. Therefore, we propose to constrain the incoming weights of each neuron to be unit-norm, which is formulated as an optimization problem over Oblique manifold. A simple yet efficient method referred to as projection based weight normalization (PBWN) is also developed to solve this problem. PBWN executes standard gradient updates, followed by projecting the updated weight back to Oblique manifold. This proposed method has the property of regularization and collaborates well with the commonly used batch normalization technique. We conduct comprehensive experiments on several widely-used image datasets including CIFAR-10, CIFAR-100, SVHN and ImageNet for supervised learning over the state-of-the-art convolutional neural networks, such as Inception, VGG and residual networks. The results show that our method is able to improve the performance of DNNs with different architectures consistently. We also apply our method to Ladder network for semi-supervised learning on permutation invariant MNIST dataset, and our method outperforms the state-of-the-art methods: we obtain test errors as 2.52%, 1.06%, and 0.91% with only 20, 50, and 100 labeled samples, respectively.

12 pages, 3 figures

References in corpus (13)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Improved Techniques for Training GANs
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• Layer Normalization
• Deep Networks with Stochastic Depth
• Triple Generative Adversarial Nets
• Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning
• On orthogonality and learning recurrent networks with long term dependencies
• Deconstructing the Ladder Network Architecture
• Neural Machine Translation with Reconstruction
• Generalized BackPropagation, Étude De Cas: Orthogonality
• DizzyRNN: Reparameterizing Recurrent Neural Networks for Norm-Preserving Backpropagation

Cited by in corpus (4)

• Normalization Techniques in Training DNNs: Methodology, Analysis and Application
• A Scale Invariant Flatness Measure for Deep Network Minima
• Feedback Gradient Descent: Efficient and Stable Optimization with Orthogonality for DNNs
• Preprint: Norm Loss: An efficient yet effective regularization method for deep neural networks