Fisher Kernel for Deep Neural Activations

Abstract

Compared to image representation based on low-level local descriptors, deep neural activations of Convolutional Neural Networks (CNNs) are richer in mid-level representation, but poorer in geometric invariance properties. In this paper, we present a straightforward framework for better image representation by combining the two approaches. To take advantages of both representations, we propose an efficient method to extract a fair amount of multi-scale dense local activations from a pre-trained CNN. We then aggregate the activations by Fisher kernel framework, which has been modified with a simple scale-wise normalization essential to make it suitable for CNN activations. Replacing the direct use of a single activation vector with our representation demonstrates significant performance improvements: +17.76 (Acc.) on MIT Indoor 67 and +7.18 (mAP) on PASCAL VOC 2007. The results suggest that our proposal can be used as a primary image representation for better performances in visual recognition tasks.

References in corpus (5)

• Caffe: Convolutional Architecture for Fast Feature Embedding
• Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
• CNN: Single-label to Multi-label
• Return of the Devil in the Details: Delving Deep into Convolutional Nets
• CNN Features off-the-shelf: an Astounding Baseline for Recognition

Cited by in corpus (10)

• Weakly Supervised Fine-Grained Image Categorization
• Hybrid CNN and Dictionary-Based Models for Scene Recognition and Domain Adaptation
• Semantic-Aware Scene Recognition
• Deep Spatial Pyramid: The Devil is Once Again in the Details
• AttentionNet: Aggregating Weak Directions for Accurate Object Detection
• Dense Image Representation with Spatial Pyramid VLAD Coding of CNN for Locally Robust Captioning
• Visual Recognition Using Directional Distribution Distance
• Backpropagation Training for Fisher Vectors within Neural Networks
• Compositional Model based Fisher Vector Coding for Image Classification
• ASK: Adaptively Selecting Key Local Features for RGB-D Scene Recognition