Self-Orthogonality Module: A Network Architecture Plug-in for Learning Orthogonal Filters
arXiv:2001.01275
Abstract
In this paper, we investigate the empirical impact of orthogonality regularization (OR) in deep learning, either solo or collaboratively. Recent works on OR showed some promising results on the accuracy. In our ablation study, however, we do not observe such significant improvement from existing OR techniques compared with the conventional training based on weight decay, dropout, and batch normalization. To identify the real gain from OR, inspired by the locality sensitive hashing (LSH) in angle estimation, we propose to introduce an implicit self-regularization into OR to push the mean and variance of filter angles in a network towards 90 and 0 simultaneously to achieve (near) orthogonality among the filters, without using any other explicit regularization. Our regularization can be implemented as an architectural plug-in and integrated with an arbitrary network. We reveal that OR helps stabilize the training process and leads to faster convergence and better generalization.
This version fixed the controversial expression in Section 2.2
References in corpus (8)
- Improving neural networks by preventing co-adaptation of feature detectors
- Stochastic Pooling for Regularization of Deep Convolutional Neural Networks
- L2 Regularization versus Batch and Weight Normalization
- Regularization for Deep Learning: A Taxonomy
- Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks
- Regularizing CNNs with Locally Constrained Decorrelations
- Generalized BackPropagation, Étude De Cas: Orthogonality
- Boosting Occluded Image Classification via Subspace Decomposition Based Estimation of Deep Features