Generalized BackPropagation, Étude De Cas: Orthogonality

Abstract

This paper introduces an extension of the backpropagation algorithm that enables us to have layers with constrained weights in a deep network. In particular, we make use of the Riemannian geometry and optimization techniques on matrix manifolds to step outside of normal practice in training deep networks, equipping the network with structures such as orthogonality or positive definiteness. Based on our development, we make another contribution by introducing the Stiefel layer, a layer with orthogonal weights. Among various applications, Stiefel layers can be used to design orthogonal filter banks, perform dimensionality reduction and feature extraction. We demonstrate the benefits of having orthogonality in deep networks through a broad set of experiments, ranging from unsupervised feature learning to fine-grained image classification.

References in corpus (3)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Fine-Grained Visual Classification of Aircraft
• Manopt, a Matlab toolbox for optimization on manifolds

Cited by in corpus (22)

• Concept Whitening for Interpretable Image Recognition
• On orthogonality and learning recurrent networks with long term dependencies
• Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group
• ABD-Net: Attentive but Diverse Person Re-Identification
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application
• Parseval Proximal Neural Networks
• Projection Based Weight Normalization for Deep Neural Networks
• Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?
• Orthogonal Relation Transforms with Graph Context Modeling for Knowledge Graph Embedding
• Orthogonal Convolutional Neural Networks
• Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training
• Information Geometry of Orthogonal Initializations and Training
• Self-Orthogonality Module: A Network Architecture Plug-in for Learning Orthogonal Filters
• Tractable Density Estimation on Learned Manifolds with Conformal Embedding Flows
• Controllable Orthogonalization in Training DNNs
• Orthogonal Wasserstein GANs
• Pruning Ternary Quantization
• Learning Bijective Feature Maps for Linear ICA
• Statistical transformer networks: learning shape and appearance models via self supervision
• Preprint: Norm Loss: An efficient yet effective regularization method for deep neural networks
• Parallelized Computation and Backpropagation Under Angle-Parametrized Orthogonal Matrices
• Simultaneously Learning Architectures and Features of Deep Neural Networks