Fisher Vectors Derived from Hybrid Gaussian-Laplacian Mixture Models for Image Annotation

Abstract

In the traditional object recognition pipeline, descriptors are densely sampled over an image, pooled into a high dimensional non-linear representation and then passed to a classifier. In recent years, Fisher Vectors have proven empirically to be the leading representation for a large variety of applications. The Fisher Vector is typically taken as the gradients of the log-likelihood of descriptors, with respect to the parameters of a Gaussian Mixture Model (GMM). Motivated by the assumption that different distributions should be applied for different datasets, we present two other Mixture Models and derive their Expectation-Maximization and Fisher Vector expressions. The first is a Laplacian Mixture Model (LMM), which is based on the Laplacian distribution. The second Mixture Model presented is a Hybrid Gaussian-Laplacian Mixture Model (HGLMM) which is based on a weighted geometric mean of the Gaussian and Laplacian distribution. An interesting property of the Expectation-Maximization algorithm for the latter is that in the maximization step, each dimension in each component is chosen to be either a Gaussian or a Laplacian. Finally, by using the new Fisher Vectors derived from HGLMMs, we achieve state-of-the-art results for both the image annotation and the image search by a sentence tasks.

arXiv commentnew version includes text synthesis by an RNN and experiments with the COCO benchmark

References in corpus (6)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Improving neural networks by preventing co-adaptation of feature detectors
• Going Deeper with Convolutions
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Explain Images with Multimodal Recurrent Neural Networks
• Deep Visual-Semantic Alignments for Generating Image Descriptions

Cited by in corpus (27)

• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Exploring Models and Data for Image Question Answering
• Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering
• Multimodal Machine Learning: A Survey and Taxonomy
• Learning like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images
• Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
• Learning Deep Structure-Preserving Image-Text Embeddings
• Word2VisualVec: Image and Video to Sentence Matching by Visual Feature Prediction
• Learning to Answer Questions From Image Using Convolutional Neural Network
• Natural Language Object Retrieval
• Learning Type-Aware Embeddings for Fashion Compatibility
• Learning Two-Branch Neural Networks for Image-Text Matching Tasks
• Learning Social Image Embedding with Deep Multimodal Attention Networks
• Multi-Cue Zero-Shot Learning with Strong Supervision
• Position Focused Attention Network for Image-Text Matching
• Coherent and Controllable Outfit Generation
• Interpretable and Globally Optimal Prediction for Textual Grounding using Image Concepts
• Deep Binaries: Encoding Semantic-Rich Cues for Efficient Textual-Visual Cross Retrieval
• Probabilistic Embeddings for Cross-Modal Retrieval
• Revisiting Cross Modal Retrieval
• Efficient Algorithms for Estimating the Parameters of Mixed Linear Regression Models
• Ladder Loss for Coherent Visual-Semantic Embedding
• On the Behavior of the Expectation-Maximization Algorithm for Mixture Models
• A Better Loss for Visual-Textual Grounding
• Backpropagation Training for Fisher Vectors within Neural Networks
• Semi Supervised Phrase Localization in a Bidirectional Caption-Image Retrieval Framework
• D-Sempre: Learning Deep Semantic-Preserving Embeddings for User interests-Social Contents Modeling