Semantic Compositional Networks for Visual Captioning
arXiv:1611.08002
Abstract
A Semantic Compositional Network (SCN) is developed for image captioning, in which semantic concepts (i.e., tags) are detected from the image, and the probability of each tag is used to compose the parameters in a long short-term memory (LSTM) network. The SCN extends each weight matrix of the LSTM to an ensemble of tag-dependent weight matrices. The degree to which each member of the ensemble is used to generate an image caption is tied to the image-dependent probability of the corresponding tag. In addition to captioning images, we also extend the SCN to generate captions for video clips. We qualitatively analyze semantic composition in SCNs, and quantitatively evaluate the algorithm on three benchmark datasets: COCO, Flickr30k, and Youtube2Text. Experimental results show that the proposed method significantly outperforms prior state-of-the-art approaches, across multiple evaluation metrics.
Accepted in CVPR 2017
References in corpus (29)
- Adam: A Method for Stochastic Optimization
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Deep Residual Learning for Image Recognition
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
- Recurrent Neural Network Regularization
- Microsoft COCO Captions: Data Collection and Evaluation Server
- Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
- Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
- Image Captioning with Semantic Attention
- Sequence to Sequence -- Video to Text
- Show and Tell: A Neural Image Caption Generator
- Learning Spatiotemporal Features with 3D Convolutional Networks
- Deep Visual-Semantic Alignments for Generating Image Descriptions
- Delving Deeper into Convolutional Networks for Learning Video Representations
- Language Models for Image Captioning: The Quirks and What Works
- Aligning where to see and what to tell: image caption with region-based attention and scene factorization
- From Captions to Visual Concepts and Back
- Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks
- Review Networks for Caption Generation
- Guiding Long-Short Term Memory for Image Caption Generation
- Attention Correctness in Neural Image Captioning
- CIDEr: Consensus-based Image Description Evaluation
- On Multiplicative Integration with Recurrent Neural Networks
- Rich Image Captioning in the Wild
- Learning Attributes Equals Multi-Source Domain Generalization
- What value do explicit high level concepts have in vision to language problems?
- Jointly Modeling Embedding and Translation to Bridge Video and Language
- Deep Compositional Captioning: Describing Novel Object Categories without Paired Training Data
- Factored Temporal Sigmoid Belief Networks for Sequence Learning
Cited by in corpus (11)
- AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks
- Bidirectional Attentive Fusion with Context Gating for Dense Video Captioning
- Hierarchical LSTM with Adjusted Temporal Attention for Video Captioning
- Video Captioning via Hierarchical Reinforcement Learning
- Hierarchical LSTMs with Adaptive Attention for Visual Captioning
- From Deterministic to Generative: Multi-Modal Stochastic RNNs for Video Captioning
- Attend and Interact: Higher-Order Object Interactions for Video Understanding
- Video Captioning with Guidance of Multimodal Latent Topics
- Better Captioning with Sequence-Level Exploration
- Grounded Objects and Interactions for Video Captioning
- MTLE: A Multitask Learning Encoder of Visual Feature Representations for Video and Movie Description