Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)

Abstract

In this paper, we present a multimodal Recurrent Neural Network (m-RNN) model for generating novel image captions. It directly models the probability distribution of generating a word given previous words and an image. Image captions are generated by sampling from this distribution. The model consists of two sub-networks: a deep recurrent neural network for sentences and a deep convolutional network for images. These two sub-networks interact with each other in a multimodal layer to form the whole m-RNN model. The effectiveness of our model is validated on four benchmark datasets (IAPR TC-12, Flickr 8K, Flickr 30K and MS COCO). Our model outperforms the state-of-the-art methods. In addition, we apply the m-RNN model to retrieval tasks for retrieving images or sentences, and achieves significant performance improvement over the state-of-the-art methods which directly optimize the ranking objective function for retrieval. The project page of this work is: www.stat.ucla.edu/~junhua.mao/m-RNN.html .

Add a simple strategy to boost the performance of image captioning task significantly. More details are shown in Section 8 of the paper. The code and related data are available at https://github.com/mjhucla/mRNN-CR ;. arXiv admin note: substantial text overlap with arXiv:1410.1090

References in corpus (8)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Sequence to Sequence Learning with Neural Networks
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Explain Images with Multimodal Recurrent Neural Networks
• Learning a Recurrent Visual Representation for Image Caption Generation
• Exploring Nearest Neighbor Approaches for Image Captioning
• CIDEr: Consensus-based Image Description Evaluation

Cited by in corpus (27)

• Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge
• Fully Connected Deep Structured Networks
• Detecting Sarcasm in Multimodal Social Platforms
• Exploring Nearest Neighbor Approaches for Image Captioning
• Context-Aware Visual Policy Network for Fine-Grained Image Captioning
• Recurrent Topic-Transition GAN for Visual Paragraph Generation
• Vision-to-Language Tasks Based on Attributes and Attention Mechanism
• A Dataset for Movie Description
• Dual Attention Networks for Multimodal Reasoning and Matching
• Comparative evaluation of CNN architectures for Image Caption Generation
• 3G structure for image caption generation
• On Distinctive Image Captioning via Comparing and Reweighting
• Long-term Multi-granularity Deep Framework for Driver Drowsiness Detection
• VQS: Linking Segmentations to Questions and Answers for Supervised Attention in VQA and Question-Focused Semantic Segmentation
• OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and Generation
• Learning Social Image Embedding with Deep Multimodal Attention Networks
• Semantic Compositional Networks for Visual Captioning
• Tell-and-Answer: Towards Explainable Visual Question Answering using Attributes and Captions
• An Effective Automatic Image Annotation Model Via Attention Model and Data Equilibrium
• Introducing Aspects of Creativity in Automatic Poetry Generation
• Sensor-Augmented Egocentric-Video Captioning with Dynamic Modal Attention
• Figure Captioning with Reasoning and Sequence-Level Training
• Auxiliary Multimodal LSTM for Audio-visual Speech Recognition and Lipreading
• Learning to Disambiguate by Asking Discriminative Questions
• Robot Object Retrieval with Contextual Natural Language Queries
• BERT-hLSTMs: BERT and Hierarchical LSTMs for Visual Storytelling
• The Long-Short Story of Movie Description