Is Second-order Information Helpful for Large-scale Visual Recognition?

Abstract

By stacking layers of convolution and nonlinearity, convolutional networks (ConvNets) effectively learn from low-level to high-level features and discriminative representations. Since the end goal of large-scale recognition is to delineate complex boundaries of thousands of classes, adequate exploration of feature distributions is important for realizing full potentials of ConvNets. However, state-of-the-art works concentrate only on deeper or wider architecture design, while rarely exploring feature statistics higher than first-order. We take a step towards addressing this problem. Our method consists in covariance pooling, instead of the most commonly used first-order pooling, of high-level convolutional features. The main challenges involved are robust covariance estimation given a small sample of large-dimensional features and usage of the manifold structure of covariance matrices. To address these challenges, we present a Matrix Power Normalized Covariance (MPN-COV) method. We develop forward and backward propagation formulas regarding the nonlinear matrix functions such that MPN-COV can be trained end-to-end. In addition, we analyze both qualitatively and quantitatively its advantage over the well-known Log-Euclidean metric. On the ImageNet 2012 validation set, by combining MPN-COV we achieve over 4%, 3% and 2.5% gains for AlexNet, VGG-M and VGG-16, respectively; integration of MPN-COV into 50-layer ResNet outperforms ResNet-101 and is comparable to ResNet-152. The source code will be available on the project page: http://www.peihuali.org/MPN-COV

accepted to ICCV 2017

References in corpus (2)

• Return of the Devil in the Details: Delving Deep into Convolutional Nets
• Non-Euclidean statistics for covariance matrices, with applications to diffusion tensor imaging

Cited by in corpus (12)

• A Two-Stage Attentive Network for Single Image Super-Resolution
• WIDER Face and Pedestrian Challenge 2018: Methods and Results
• Fixing the train-test resolution discrepancy
• $A^2$-Nets: Double Attention Networks
• Multi-scale Location-aware Kernel Representation for Object Detection
• Statistically Motivated Second Order Pooling
• Learning a Robust Representation via a Deep Network on Symmetric Positive Definite Manifolds
• MoNet: Moments Embedding Network
• W2S: Microscopy Data with Joint Denoising and Super-Resolution for Widefield to SIM Mapping
• Compact Approximation for Polynomial of Covariance Feature
• Kernelized Classification in Deep Networks
• DeepSquare: Boosting the Learning Power of Deep Convolutional Neural Networks with Elementwise Square Operators