Low-rank Bilinear Pooling for Fine-Grained Classification

Abstract

Pooling second-order local feature statistics to form a high-dimensional bilinear feature has been shown to achieve state-of-the-art performance on a variety of fine-grained classification tasks. To address the computational demands of high feature dimensionality, we propose to represent the covariance features as a matrix and apply a low-rank bilinear classifier. The resulting classifier can be evaluated without explicitly computing the bilinear feature map which allows for a large reduction in the compute time as well as decreasing the effective number of parameters to be learned. To further compress the model, we propose classifier co-decomposition that factorizes the collection of bilinear classifiers into a common factor and compact per-class terms. The co-decomposition idea can be deployed through two convolutional layers and trained in an end-to-end architecture. We suggest a simple yet effective initialization that avoids explicitly first training and factorizing the larger bilinear classifiers. Through extensive experiments, we show that our model achieves state-of-the-art performance on several public datasets for fine-grained classification trained with only category labels. Importantly, our final model is an order of magnitude smaller than the recently proposed compact bilinear model, and three orders smaller than the standard bilinear CNN model.

References in corpus (10)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Fine-Grained Visual Classification of Aircraft
• The Loss Surfaces of Multilayer Networks
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• Random Feature Maps for Dot Product Kernels
• Fine-grained pose prediction, normalization, and recognition
• Part Detector Discovery in Deep Convolutional Neural Networks
• Fine-grained Categorization and Dataset Bootstrapping using Deep Metric Learning with Humans in the Loop
• The Unreasonable Effectiveness of Noisy Data for Fine-Grained Recognition
• Part-Stacked CNN for Fine-Grained Visual Categorization

Cited by in corpus (4)

• Fully Convolutional Attention Networks for Fine-Grained Recognition
• Saliency for Fine-grained Object Recognition in Domains with Scarce Training Data
• Dynamic Computational Time for Visual Attention
• Fine-Grained Representation Learning and Recognition by Exploiting Hierarchical Semantic Embedding