Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)

Abstract

In this paper, we present a multimodal Recurrent Neural Network (m-RNN) model for generating novel image captions. It directly models the probability distribution of generating a word given previous words and an image. Image captions are generated by sampling from this distribution. The model consists of two sub-networks: a deep recurrent neural network for sentences and a deep convolutional network for images. These two sub-networks interact with each other in a multimodal layer to form the whole m-RNN model. The effectiveness of our model is validated on four benchmark datasets (IAPR TC-12, Flickr 8K, Flickr 30K and MS COCO). Our model outperforms the state-of-the-art methods. In addition, we apply the m-RNN model to retrieval tasks for retrieving images or sentences, and achieves significant performance improvement over the state-of-the-art methods which directly optimize the ranking objective function for retrieval. The project page of this work is: www.stat.ucla.edu/~junhua.mao/m-RNN.html .

arXiv commentAdd a simple strategy to boost the performance of image captioning task significantly. More details are shown in Section 8 of the paper. The code and related data are available at https://github.com/mjhucla/mRNN-CR ;. arXiv admin note: substantial text overlap with arXiv:1410.1090

References in corpus (11)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Sequence to Sequence Learning with Neural Networks
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Explain Images with Multimodal Recurrent Neural Networks
• Learning a Recurrent Visual Representation for Image Caption Generation
• Exploring Nearest Neighbor Approaches for Image Captioning
• Language Models for Image Captioning: The Quirks and What Works
• From Captions to Visual Concepts and Back
• Learning like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images
• CIDEr: Consensus-based Image Description Evaluation

Cited by in corpus (81)

• Session-based Recommendation with Graph Neural Networks
• Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge
• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
• An efficient framework for learning sentence representations
• Fully Connected Deep Structured Networks
• Visual Question Answering: Datasets, Algorithms, and Future Challenges
• Image Captioning with Semantic Attention
• Detecting Sarcasm in Multimodal Social Platforms
• Sequence to Sequence -- Video to Text
• Exploring Nearest Neighbor Approaches for Image Captioning
• Context-Aware Visual Policy Network for Fine-Grained Image Captioning
• Learning Contextual Dependencies with Convolutional Hierarchical Recurrent Neural Networks
• Context-Aware Visual Policy Network for Sequence-Level Image Captioning
• A Survey on Multi-View Clustering
• Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequences
• Where to put the Image in an Image Caption Generator
• Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks
• Recurrent Topic-Transition GAN for Visual Paragraph Generation
• CNN+CNN: Convolutional Decoders for Image Captioning
• Multimodal Convolutional Neural Networks for Matching Image and Sentence
• Vision-to-Language Tasks Based on Attributes and Attention Mechanism
• A Dataset for Movie Description
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• Video Storytelling: Textual Summaries for Events
• Dual Attention Networks for Multimodal Reasoning and Matching
• Cascaded Revision Network for Novel Object Captioning
• Visual Question Answering for Cultural Heritage
• SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioning
• Look, Imagine and Match: Improving Textual-Visual Cross-Modal Retrieval with Generative Models
• Comparative evaluation of CNN architectures for Image Caption Generation
• 3G structure for image caption generation
• Creativity: Generating Diverse Questions using Variational Autoencoders
• On Distinctive Image Captioning via Comparing and Reweighting
• Long-term Multi-granularity Deep Framework for Driver Drowsiness Detection
• Speaking the Same Language: Matching Machine to Human Captions by Adversarial Training
• Regularizing RNNs for Caption Generation by Reconstructing The Past with The Present
• Weakly Supervised Dense Video Captioning
• Natural Language Object Retrieval
• VQS: Linking Segmentations to Questions and Answers for Supervised Attention in VQA and Question-Focused Semantic Segmentation
• Learning with hidden variables
• OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and Generation
• Forecasting the Spread of Covid-19 Under Control Scenarios Using LSTM and Dynamic Behavioral Models
• Multi-Cue Zero-Shot Learning with Strong Supervision
• Learning Social Image Embedding with Deep Multimodal Attention Networks
• Semantic Compositional Networks for Visual Captioning
• Generating Descriptions with Grounded and Co-Referenced People
• Face2Text: Collecting an Annotated Image Description Corpus for the Generation of Rich Face Descriptions
• Skeleton Key: Image Captioning by Skeleton-Attribute Decomposition
• Tell-and-Answer: Towards Explainable Visual Question Answering using Attributes and Captions
• A Restricted Visual Turing Test for Deep Scene and Event Understanding
• Lie Access Neural Turing Machine
• An Effective Automatic Image Annotation Model Via Attention Model and Data Equilibrium
• Unpaired Image Captioning by Language Pivoting
• Stacked Cross-modal Feature Consolidation Attention Networks for Image Captioning
• AutoCaption: Image Captioning with Neural Architecture Search
• Introducing Aspects of Creativity in Automatic Poetry Generation
• Sensor-Augmented Egocentric-Video Captioning with Dynamic Modal Attention
• Aligning Multilingual Word Embeddings for Cross-Modal Retrieval Task
• Scene Graph Parsing as Dependency Parsing
• End-to-End Attention-based Image Captioning
• SNN: Stacked Neural Networks
• Figure Captioning with Reasoning and Sequence-Level Training
• Auxiliary Multimodal LSTM for Audio-visual Speech Recognition and Lipreading
• Variational Prefix Tuning for Diverse and Accurate Code Summarization Using Pre-trained Language Models
• Learning to Disambiguate by Asking Discriminative Questions
• Order embeddings and character-level convolutions for multimodal alignment
• Spatial Memory for Context Reasoning in Object Detection
• Intention Oriented Image Captions with Guiding Objects
• When deep learning meets causal inference: a computational framework for drug repurposing from real-world data
• A sequential guiding network with attention for image captioning
• An Overview of Cross-media Retrieval: Concepts, Methodologies, Benchmarks and Challenges
• simNet: Stepwise Image-Topic Merging Network for Generating Detailed and Comprehensive Image Captions
• Discriminative Bimodal Networks for Visual Localization and Detection with Natural Language Queries
• Improving Classification by Improving Labelling: Introducing Probabilistic Multi-Label Object Interaction Recognition
• Robot Object Retrieval with Contextual Natural Language Queries
• BERT-hLSTMs: BERT and Hierarchical LSTMs for Visual Storytelling
• Conditionally Learn to Pay Attention for Sequential Visual Task
• Scene-based Factored Attention for Image Captioning
• The Long-Short Story of Movie Description
• Dropout during inference as a model for neurological degeneration in an image captioning network
• Multimodal Story Generation on Plural Images