Explain Images with Multimodal Recurrent Neural Networks
arXiv:1410.1090
Abstract
In this paper, we present a multimodal Recurrent Neural Network (m-RNN) model for generating novel sentence descriptions to explain the content of images. It directly models the probability distribution of generating a word given previous words and the image. Image descriptions are generated by sampling from this distribution. The model consists of two sub-networks: a deep recurrent neural network for sentences and a deep convolutional network for images. These two sub-networks interact with each other in a multimodal layer to form the whole m-RNN model. The effectiveness of our model is validated on three benchmark datasets (IAPR TC-12, Flickr 8K, and Flickr 30K). Our model outperforms the state-of-the-art generative method. In addition, the m-RNN model can be applied to retrieval tasks for retrieving images or sentences, and achieves significant performance improvement over the state-of-the-art methods which directly optimize the ranking objective function for retrieval.
References in corpus (2)
Cited by in corpus (70)
- Microsoft COCO Captions: Data Collection and Evaluation Server
- Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
- Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge
- Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
- Exploring Models and Data for Image Question Answering
- Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
- Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering
- Learning a Recurrent Visual Representation for Image Caption Generation
- Exploring Nearest Neighbor Approaches for Image Captioning
- Deep Visual-Semantic Alignments for Generating Image Descriptions
- Language Models for Image Captioning: The Quirks and What Works
- Aligning where to see and what to tell: image caption with region-based attention and scene factorization
- Where to put the Image in an Image Caption Generator
- From Captions to Visual Concepts and Back
- Recurrent Neural Networks Hardware Implementation on FPGA
- DenseCap: Fully Convolutional Localization Networks for Dense Captioning
- Fisher Vectors Derived from Hybrid Gaussian-Laplacian Mixture Models for Image Annotation
- Towards Goal-Oriented Semantic Signal Processing: Applications and Future Challenges
- Generating Visual Explanations
- Learning like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images
- Multimodal Convolutional Neural Networks for Matching Image and Sentence
- Learning to generalize to new compositions in image understanding
- Natural Language Understanding with Distributed Representation
- Phrase-based Image Captioning
- From Images to Sentences through Scene Description Graphs using Commonsense Reasoning and Knowledge
- Multimedia Semantic Integrity Assessment Using Joint Embedding Of Images And Text
- Boosting Image Captioning with Attributes
- Yin and Yang: Balancing and Answering Binary Visual Questions
- Identity-Aware Textual-Visual Matching with Latent Co-attention
- Learning to Read Chest X-Rays: Recurrent Neural Cascade Model for Automated Image Annotation
- Jointly Modeling Embedding and Translation to Bridge Video and Language
- Simple Image Description Generator via a Linear Phrase-Based Approach
- Learning to Answer Questions From Image Using Convolutional Neural Network
- A Pooling Approach to Modelling Spatial Relations for Image Retrieval and Annotation
- Learning Semantic Concepts and Order for Image and Sentence Matching
- Annotation Order Matters: Recurrent Image Annotator for Arbitrary Length Image Tagging
- Content-based Video Indexing and Retrieval Using Corr-LDA
- Doubly Attentive Transformer Machine Translation
- Neural Self Talk: Image Understanding via Continuous Questioning and Answering
- Video Captioning with Transferred Semantic Attributes
- Generating Multi-Sentence Lingual Descriptions of Indoor Scenes
- Automatic Concept Discovery from Parallel Text and Visual Corpora
- MirBot: A collaborative object recognition system for smartphones using convolutional neural networks
- A Semi-supervised Framework for Image Captioning
- A Comprehensive Survey of Deep Learning for Image Captioning
- A New Evaluation Protocol and Benchmarking Results for Extendable Cross-media Retrieval
- X-Linear Attention Networks for Image Captioning
- Multilingual Visual Sentiment Concept Matching
- Stacked Cross-modal Feature Consolidation Attention Networks for Image Captioning
- Instance-aware Image and Sentence Matching with Selective Multimodal LSTM
- Multi-Modal Music Information Retrieval: Augmenting Audio-Analysis with Visual Computing for Improved Music Video Analysis
- Dense Captioning with Joint Inference and Visual Context
- Exploring Overall Contextual Information for Image Captioning in Human-Like Cognitive Style
- Conditional Text Generation for Harmonious Human-Machine Interaction
- Commonly Uncommon: Semantic Sparsity in Situation Recognition
- DeepDiary: Automatic Caption Generation for Lifelogging Image Streams
- Transfer learning from language models to image caption generators: Better models may not transfer better
- Heterogeneous Contrastive Learning
- Geometry-Entangled Visual Semantic Transformer for Image Captioning
- Improving Image Captioning by Leveraging Knowledge Graphs
- Kite: Automatic speech recognition for unmanned aerial vehicles
- Generate Image Descriptions based on Deep RNN and Memory Cells for Images Features
- On Architectures for Including Visual Information in Neural Language Models for Image Description
- Event Specific Multimodal Pattern Mining with Image-Caption Pairs
- Visuallly Grounded Generation of Entailments from Premises
- Automated Image Captioning for Rapid Prototyping and Resource Constrained Environments
- Stack-VS: Stacked Visual-Semantic Attention for Image Caption Generation
- Watch and Learn: Mapping Language and Noisy Real-world Videos with Self-supervision
- Characterization and recognition of handwritten digits using Julia
- AI-Powered Text Generation for Harmonious Human-Machine Interaction: Current State and Future Directions