Fisher Vectors Derived from Hybrid Gaussian-Laplacian Mixture Models for Image Annotation

Abstract

In the traditional object recognition pipeline, descriptors are densely sampled over an image, pooled into a high dimensional non-linear representation and then passed to a classifier. In recent years, Fisher Vectors have proven empirically to be the leading representation for a large variety of applications. The Fisher Vector is typically taken as the gradients of the log-likelihood of descriptors, with respect to the parameters of a Gaussian Mixture Model (GMM). Motivated by the assumption that different distributions should be applied for different datasets, we present two other Mixture Models and derive their Expectation-Maximization and Fisher Vector expressions. The first is a Laplacian Mixture Model (LMM), which is based on the Laplacian distribution. The second Mixture Model presented is a Hybrid Gaussian-Laplacian Mixture Model (HGLMM) which is based on a weighted geometric mean of the Gaussian and Laplacian distribution. An interesting property of the Expectation-Maximization algorithm for the latter is that in the maximization step, each dimension in each component is chosen to be either a Gaussian or a Laplacian. Finally, by using the new Fisher Vectors derived from HGLMMs, we achieve state-of-the-art results for both the image annotation and the image search by a sentence tasks.

new version includes text synthesis by an RNN and experiments with the COCO benchmark

References in corpus (6)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Improving neural networks by preventing co-adaptation of feature detectors
• Going Deeper with Convolutions
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Explain Images with Multimodal Recurrent Neural Networks
• Deep Visual-Semantic Alignments for Generating Image Descriptions

Cited by in corpus (5)

• Learning Social Image Embedding with Deep Multimodal Attention Networks
• Position Focused Attention Network for Image-Text Matching
• Deep Binaries: Encoding Semantic-Rich Cues for Efficient Textual-Visual Cross Retrieval
• Ladder Loss for Coherent Visual-Semantic Embedding
• Backpropagation Training for Fisher Vectors within Neural Networks