Generating Descriptions for Sequential Images with Local-Object Attention and Global Semantic Context Modelling
arXiv:2012.01295
Abstract
In this paper, we propose an end-to-end CNN-LSTM model for generating descriptions for sequential images with a local-object attention mechanism. To generate coherent descriptions, we capture global semantic context using a multi-layer perceptron, which learns the dependencies between sequential images. A paralleled LSTM network is exploited for decoding the sequence descriptions. Experimental results show that our model outperforms the baseline across three different evaluation metrics on the datasets published by Microsoft.
arXiv commentAccepted by INLG 2018
References in corpus (8)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Neural Machine Translation by Jointly Learning to Align and Translate
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Image Captioning with Semantic Attention
- Show and Tell: A Neural Image Caption Generator
- Visual Storytelling
- Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning
- Learning Visual Storylines with Skipping Recurrent Neural Networks