Incorporating Textual Evidence in Visual Storytelling

Abstract

Previous work on visual storytelling mainly focused on exploring image sequence as evidence for storytelling and neglected textual evidence for guiding story generation. Motivated by human storytelling process which recalls stories for familiar images, we exploit textual evidence from similar images to help generate coherent and meaningful stories. To pick the images which may provide textual experience, we propose a two-step ranking method based on image object recognition techniques. To utilize textual information, we design an extended Seq2Seq model with two-channel encoder and attention. Experiments on the VIST dataset show that our method outperforms state-of-the-art baseline models without heavy engineering.

References in corpus (9)

• Deep Residual Learning for Image Recognition
• Effective Approaches to Attention-based Neural Machine Translation
• Visual Storytelling
• GLAC Net: GLocal Attention Cascading Networks for Multi-image Cued Story Generation
• Contextualize, Show and Tell: A Neural Visual Storyteller
• Stories for Images-in-Sequence by using Visual and Narrative Components
• OBJ2TEXT: Generating Visually Descriptive Language from Object Layouts
• Using Deep Object Features for Image Descriptions
• simNet: Stepwise Image-Topic Merging Network for Generating Detailed and Comprehensive Image Captions