Where to Focus: Deep Attention-based Spatially Recurrent Bilinear Networks for Fine-Grained Visual Recognition

Abstract

Fine-grained visual recognition typically depends on modeling subtle difference from object parts. However, these parts often exhibit dramatic visual variations such as occlusions, viewpoints, and spatial transformations, making it hard to detect. In this paper, we present a novel attention-based model to automatically, selectively and accurately focus on critical object regions with higher importance against appearance variations. Given an image, two different Convolutional Neural Networks (CNNs) are constructed, where the outputs of two CNNs are correlated through bilinear pooling to simultaneously focus on discriminative regions and extract relevant features. To capture spatial distributions among the local regions with visual attention, soft attention based spatial Long-Short Term Memory units (LSTMs) are incorporated to realize spatially recurrent yet visually selective over local input patterns. All the above intuitions equip our network with the following novel model: two-stream CNN layers, bilinear pooling layer, spatial recurrent layer with location attention are jointly trained via an end-to-end fashion to serve as the part detector and feature extractor, whereby relevant features are localized and extracted attentively. We show the significance of our network against two well-known visual recognition tasks: fine-grained image classification and person re-identification.

arXiv comment8 pages

References in corpus (12)

• Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
• Fine-Grained Visual Classification of Aircraft
• Return of the Devil in the Details: Delving Deep into Convolutional Nets
• Multi-View Spectral Clustering via Structured Low-Rank Matrix Factorization
• Bird Species Categorization Using Pose Normalized Deep Convolutional Nets
• PersonNet: Person Re-identification with Deep Convolutional Neural Networks
• Deep Adaptive Feature Embedding with Local Sample Distributions for Person Re-identification
• Effective Multi-Query Expansions: Collaborative Deep Networks for Robust Landmark Retrieval
• What-and-Where to Match: Deep Spatially Multiplicative Integration Networks for Person Re-identification
• Structured Deep Hashing with Convolutional Neural Networks for Fast Person Re-identification
• Multi-Dimensional Recurrent Neural Networks
• Beyond Low-Rank Representations: Orthogonal Clustering Basis Reconstruction with Optimized Graph Structure for Multi-view Spectral Clustering