Improved Bilinear Pooling with CNNs

Abstract

Bilinear pooling of Convolutional Neural Network (CNN) features [22, 23], and their compact variants [10], have been shown to be effective at fine-grained recognition, scene categorization, texture recognition, and visual question-answering tasks among others. The resulting representation captures second-order statistics of convolutional features in a translationally invariant manner. In this paper we investigate various ways of normalizing these statistics to improve their representation power. In particular we find that the matrix square-root normalization offers significant improvements and outperforms alternative schemes such as the matrix logarithm normalization when combined with elementwise square-root and l2 normalization. This improves the accuracy by 2-3% on a range of fine-grained recognition datasets leading to a new state of the art. We also investigate how the accuracy of matrix function computations effect network training and evaluation. In particular we compare against a technique for estimating matrix square-root gradients via solving a Lyapunov equation that is more numerically accurate than computing gradients via a Singular Value Decomposition (SVD). We find that while SVD gradients are numerically inaccurate the overall effect on the final accuracy is negligible once boundary cases are handled carefully. We present an alternative scheme for computing gradients that is faster and yet it offers improvements over the baseline model. Finally we show that the matrix square-root computed approximately using a few Newton iterations is just as accurate for the classification task but allows an order-of-magnitude faster GPU implementation compared to SVD decomposition.

References in corpus (2)

• Fine-Grained Visual Classification of Aircraft
• Return of the Devil in the Details: Delving Deep into Convolutional Nets

Cited by in corpus (11)

• Feature Fusion Vision Transformer for Fine-Grained Visual Categorization
• Unsupervised Learning of Dense Optical Flow, Depth and Egomotion from Sparse Event Data
• Chest X-rays Classification: A Multi-Label and Fine-Grained Problem
• Hierarchical Bilinear Pooling for Fine-Grained Visual Recognition
• Statistically Motivated Second Order Pooling
• Deep Attentional Structured Representation Learning for Visual Recognition
• Second-order Democratic Aggregation
• Local Temporal Bilinear Pooling for Fine-grained Action Parsing
• MoNet: Moments Embedding Network
• Approximated Bilinear Modules for Temporal Modeling
• REAPS: Towards Better Recognition of Fine-grained Images by Region Attending and Part Sequencing