Stories for Images-in-Sequence by using Visual and Narrative Components
arXiv:1805.05622 · doi:10.1007/978-3-030-00825-3_13
Abstract
Recent research in AI is focusing towards generating narrative stories about visual scenes. It has the potential to achieve more human-like understanding than just basic description generation of images- in-sequence. In this work, we propose a solution for generating stories for images-in-sequence that is based on the Sequence to Sequence model. As a novelty, our encoder model is composed of two separate encoders, one that models the behaviour of the image sequence and other that models the sentence-story generated for the previous image in the sequence of images. By using the image sequence encoder we capture the temporal dependencies between the image sequence and the sentence-story and by using the previous sentence-story encoder we achieve a better story flow. Our solution generates long human-like stories that not only describe the visual context of the image sequence but also contains narrative and evaluative language. The obtained results were confirmed by manual human evaluation.
arXiv comment12 pages, 4 figures, ICT Innovations 2018
References in corpus (14)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Sequence to Sequence Learning with Neural Networks
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
- On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
- VQA: Visual Question Answering
- Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering
- Using Descriptive Video Services to Create a Large Data Source for Video Annotation Research
- Deep Visual-Semantic Alignments for Generating Image Descriptions
- Visual Storytelling
- A Dataset for Movie Description
- Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures