Semantic Compositional Networks for Visual Captioning

Abstract

A Semantic Compositional Network (SCN) is developed for image captioning, in which semantic concepts (i.e., tags) are detected from the image, and the probability of each tag is used to compose the parameters in a long short-term memory (LSTM) network. The SCN extends each weight matrix of the LSTM to an ensemble of tag-dependent weight matrices. The degree to which each member of the ensemble is used to generate an image caption is tied to the image-dependent probability of the corresponding tag. In addition to captioning images, we also extend the SCN to generate captions for video clips. We qualitatively analyze semantic composition in SCNs, and quantitatively evaluate the algorithm on three benchmark datasets: COCO, Flickr30k, and Youtube2Text. Experimental results show that the proposed method significantly outperforms prior state-of-the-art approaches, across multiple evaluation metrics.

Accepted in CVPR 2017

References in corpus (10)

• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Recurrent Neural Network Regularization
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
• Show and Tell: A Neural Image Caption Generator
• Deep Visual-Semantic Alignments for Generating Image Descriptions
• Aligning where to see and what to tell: image caption with region-based attention and scene factorization
• CIDEr: Consensus-based Image Description Evaluation
• Jointly Modeling Embedding and Translation to Bridge Video and Language

Cited by in corpus (4)

• AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks
• Hierarchical LSTM with Adjusted Temporal Attention for Video Captioning
• From Deterministic to Generative: Multi-Modal Stochastic RNNs for Video Captioning
• Grounded Objects and Interactions for Video Captioning