Deep CNNs Meet Global Covariance Pooling: Better Representation and Generalization

Abstract

Compared with global average pooling in existing deep convolutional neural networks (CNNs), global covariance pooling can capture richer statistics of deep features, having potential for improving representation and generalization abilities of deep CNNs. However, integration of global covariance pooling into deep CNNs brings two challenges: (1) robust covariance estimation given deep features of high dimension and small sample size; (2) appropriate usage of geometry of covariances. To address these challenges, we propose a global Matrix Power Normalized COVariance (MPN-COV) Pooling. Our MPN-COV conforms to a robust covariance estimator, very suitable for scenario of high dimension and small sample size. It can also be regarded as Power-Euclidean metric between covariances, effectively exploiting their geometry. Furthermore, a global Gaussian embedding network is proposed to incorporate first-order statistics into MPN-COV. For fast training of MPN-COV networks, we implement an iterative matrix square root normalization, avoiding GPU unfriendly eigen-decomposition inherent in MPN-COV. Additionally, progressive 1x1 convolutions and group convolution are introduced to compress covariance representations. The proposed methods are highly modular, readily plugged into existing deep CNNs. Extensive experiments are conducted on large-scale object classification, scene categorization, fine-grained visual recognition and texture classification, showing our methods outperform the counterparts and obtain state-of-the-art performance.

arXiv commentAccepted to IEEE TPAMI. Code is at http://peihuali.org/MPN-COV/

References in corpus (8)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Fine-Grained Visual Classification of Aircraft
• Shrinkage Algorithms for MMSE Covariance Estimation
• Non-Euclidean statistics for covariance matrices, with applications to diffusion tensor imaging
• Training Deep Networks with Structured Layers by Matrix Backpropagation
• Deep FisherNet for Object Classification
• Towards Effective Codebookless Model for Image Classification

Cited by in corpus (9)

• Second-Order Pooling for Graph Neural Networks
• Convolutional Fine-Grained Classification with Self-Supervised Target Relation Regularization
• On the Eigenvalues of Global Covariance Pooling for Fine-grained Visual Recognition
• Fine-Grained Image Analysis with Deep Learning: A Survey
• Conditional Temporal Neural Processes with Covariance Loss
• GAN for Vision, KG for Relation: a Two-stage Deep Network for Zero-shot Action Recognition
• Learning Class Unique Features in Fine-Grained Visual Classification
• A${}^2$M${}^2$-Net: Adaptively Aligned Multi-Scale Moment for Few-Shot Action Recognition
• Kernelized Classification in Deep Networks