GLAC Net: GLocal Attention Cascading Networks for Multi-image Cued Story Generation
arXiv:1805.10973
Abstract
The task of multi-image cued story generation, such as visual storytelling dataset (VIST) challenge, is to compose multiple coherent sentences from a given sequence of images. The main difficulty is how to generate image-specific sentences within the context of overall images. Here we propose a deep learning network model, GLAC Net, that generates visual stories by combining global-local (glocal) attention and context cascading mechanisms. The model incorporates two levels of attention, i.e., overall encoding level and image feature level, to construct image-dependent sentences. While standard attention configuration needs a large number of parameters, the GLAC Net implements them in a very simple way via hard connections from the outputs of encoders or image features onto the sentence generators. The coherency of the generated story is further improved by conveying (cascading) the information of the previous sentence to the next sentence serially. We evaluate the performance of the GLAC Net on the visual storytelling dataset (VIST) and achieve very competitive results compared to the state-of-the-art techniques. Our code and pre-trained models are available here.
arXiv comment6 pages, 3 figures, paper for Visual Storytelling Challenge in Storytelling Workshop co-located with NAACL 2018, source code and pre-trained models are available at https://github.com/tkim-snu/GLACNet
References in corpus (13)
- Distributed Representations of Words and Phrases and their Compositionality
- Sequence to Sequence Learning with Neural Networks
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Teaching Machines to Read and Comprehend
- Generative Adversarial Text to Image Synthesis
- Convolutional Neural Networks for Sentence Classification
- VQA: Visual Question Answering
- Multimodal Residual Learning for Visual QA
- Comparing deep neural networks against humans: object recognition when the signal gets weaker
- Visual Storytelling
- A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories
- Story Generation from Sequence of Independent Short Descriptions
Cited by in corpus (8)
- Informative Visual Storytelling with Cross-modal Rules
- "My Way of Telling a Story": Persona based Grounded Story Generation
- Induction and Reference of Entities in a Visual Story
- Knowledge-Enriched Visual Storytelling
- Incorporating Textual Evidence in Visual Storytelling
- Character-Centric Storytelling
- Plot and Rework: Modeling Storylines for Visual Storytelling
- On How Users Edit Computer-Generated Visual Stories