Training Deep Networks with Structured Layers by Matrix Backpropagation

Abstract

Deep neural network architectures have recently produced excellent results in a variety of areas in artificial intelligence and visual recognition, well surpassing traditional shallow architectures trained using hand-designed features. The power of deep networks stems both from their ability to perform local computations followed by pointwise non-linearities over increasingly larger receptive fields, and from the simplicity and scalability of the gradient-descent training procedure based on backpropagation. An open problem is the inclusion of layers that perform global, structured matrix computations like segmentation (e.g. normalized cuts) or higher-order pooling (e.g. log-tangent space metrics defined over the manifold of symmetric positive definite matrices) while preserving the validity and efficiency of an end-to-end deep training framework. In this paper we propose a sound mathematical apparatus to formally integrate global structured computation into deep computation architectures. At the heart of our methodology is the development of the theory and practice of backpropagation that generalizes to the calculus of adjoint matrix variations. The proposed matrix backpropagation methodology applies broadly to a variety of problems in machine learning or computational perception. Here we illustrate it by performing visual segmentation experiments using the BSDS and MSCOCO benchmarks, where we show that deep networks relying on second-order pooling and normalized cuts layers, trained end-to-end using matrix backpropagation, outperform counterparts that do not take advantage of such global layers.

This is an extended version of our ICCV 2015 article

References in corpus (9)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Deep Learning in Neural Networks: An Overview
• Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs
• Deep Convolutional Networks on Graph-Structured Data
• Fully Convolutional Networks for Semantic Segmentation
• Convolutional Feature Masking for Joint Object and Stuff Segmentation
• Fully Connected Deep Structured Networks
• Simultaneous Detection and Segmentation
• Distributed optimization of deeply nested systems

Cited by in corpus (20)

• Differentiable Programming Tensor Networks
• Supervised Fitting of Geometric Primitives to 3D Point Clouds
• Meta-learning with differentiable closed-form solvers
• Deep CNNs Meet Global Covariance Pooling: Better Representation and Generalization
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application
• COCO-Stuff: Thing and Stuff Classes in Context
• An Analysis of SVD for Deep Rotation Estimation
• Is Second-order Information Helpful for Large-scale Visual Recognition?
• Towards Faster Training of Global Covariance Pooling Networks by Iterative Matrix Square Root Normalization
• Switchable Whitening for Deep Representation Learning
• Grouping-matrix based Graph Pooling with Adaptive Number of Clusters
• Statistically Motivated Second Order Pooling
• QR and LQ Decomposition Matrix Backpropagation Algorithms for Square, Wide, and Deep -- Real or Complex -- Matrices and Their Software Implementation
• What Deep CNNs Benefit from Global Covariance Pooling: An Optimization Perspective
• Controllable Orthogonalization in Training DNNs
• Deep, Dense, and Low-Rank Gaussian Conditional Random Fields
• Grassmann Iterative Linear Discriminant Analysis with Proxy Matrix Optimization
• A Simple Riemannian Manifold Network for Image Set Classification
• Deep Graph Matching under Quadratic Constraint
• SVMax: A Feature Embedding Regularizer