Visual Storytelling

Abstract

We introduce the first dataset for sequential vision-to-language, and explore how this data may be used for the task of visual storytelling. The first release of this dataset, SIND v.1, includes 81,743 unique photos in 20,211 sequences, aligned to both descriptive (caption) and story language. We establish several strong baselines for the storytelling task, and motivate an automatic metric to benchmark progress. Modelling concrete description as well as figurative and social language, as provided in this dataset and the storytelling task, has the potential to move artificial intelligence from basic understandings of typical visual scenes towards more and more human-like understanding of grounded event structure and subjective expression.

to appear in NAACL 2016

Cited by in corpus (56)

• The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes
• Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generation
• Recurrent Topic-Transition GAN for Visual Paragraph Generation
• GLAC Net: GLocal Attention Cascading Networks for Multi-image Cued Story Generation
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• Plan-And-Write: Towards Better Automatic Storytelling
• No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytelling
• MemexQA: Visual Memex Question Answering
• Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigation
• StoryGAN: A Sequential Conditional GAN for Story Visualization
• Hierarchically Structured Reinforcement Learning for Topically Coherent Visual Story Generation
• Attacking Visual Language Grounding with Adversarial Examples: A Case Study on Neural Image Captioning
• Using Inter-Sentence Diverse Beam Search to Reduce Redundancy in Visual Storytelling
• Stories for Images-in-Sequence by using Visual and Narrative Components
• A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation
• Strategies for Structuring Story Generation
• Generating Descriptions with Grounded and Co-Referenced People
• Learning to Globally Edit Images with Textual Description
• Graph-based Multi-hop Reasoning for Long Text Generation
• A Benchmark of Visual Storytelling in Social Media
• Unsupervised Visual-Linguistic Reference Resolution in Instructional Videos
• "My Way of Telling a Story": Persona based Grounded Story Generation
• Conversational Exploratory Search via Interactive Storytelling
• Video Highlight Prediction Using Audience Chat Reactions
• Neural Sentence Ordering Based on Constraint Graphs
• Induction and Reference of Entities in a Visual Story
• What Makes A Good Story? Designing Composite Rewards for Visual Storytelling
• Curiosity-driven Reinforcement Learning for Diverse Visual Paragraph Generation
• A System for Automated Image Editing from Natural Language Commands
• Local and Global Context-Based Pairwise Models for Sentence Ordering
• Visual Forecasting by Imitating Dynamics in Natural Sequences
• Visual Storytelling via Predicting Anchor Word Embeddings in the Stories
• Knowledge-driven Encode, Retrieve, Paraphrase for Medical Image Report Generation
• Deep Multi-Modal Image Correspondence Learning
• A Skeleton-Based Model for Promoting Coherence Among Sentences in Narrative Story Generation
• Keep it Consistent: Topic-Aware Storytelling from an Image Stream via Iterative Multi-agent Communication
• A Pipeline for Creative Visual Storytelling
• Dixit: Interactive Visual Storytelling via Term Manipulation
• Topological Sort for Sentence Ordering
• Hierarchical Photo-Scene Encoder for Album Storytelling
• A Memory Network Approach for Story-based Temporal Summarization of 360° Videos
• Evaluating Text-to-Image Matching using Binary Image Selection (BISON)
• EVOQUER: Enhancing Temporal Grounding with Video-Pivoted BackQuery Generation
• The Amazing Mysteries of the Gutter: Drawing Inferences Between Panels in Comic Book Narratives
• Incorporating Textual Evidence in Visual Storytelling
• Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning
• Hide-and-Tell: Learning to Bridge Photo Streams for Visual Storytelling
• An Unsupervised Sampling Approach for Image-Sentence Matching Using Document-Level Structural Information
• Mining Commonsense Facts from the Physical World
• Towards Understanding End-of-trip Instructions in a Taxi Ride Scenario
• Transformation Driven Visual Reasoning
• Generating Descriptions for Sequential Images with Local-Object Attention and Global Semantic Context Modelling
• Building A User-Centric and Content-Driven Socialbot
• Learning to Sort Image Sequences via Accumulated Temporal Differences
• Customized Image Narrative Generation via Interactive Visual Question Generation and Answering
• On How Users Edit Computer-Generated Visual Stories