ViNTER: Image Narrative Generation with Emotion-Arc-Aware Transformer

Abstract

Image narrative generation is a task to create a story from an image with a subjective viewpoint. Given the importance of the subjective feelings of writers, readers, and characters in storytelling, an image narrative generation method should consider human emotion. In this study, we propose a novel method of image narrative generation called ViNTER (Visual Narrative Transformer with Emotion arc Representation), which takes "emotion arc" as input to capture a sequence of emotional changes. Since emotion arcs represent the trajectory of emotional change, it is expected that we can include detailed information about the emotional changes in the story to the model. We present experimental results of both automatic and manual evaluations on the Image Narrative dataset and demonstrate the effectiveness of the proposed approach.

References in corpus (8)

• Neural Machine Translation by Jointly Learning to Align and Translate
• Decoupled Weight Decay Regularization
• BERTScore: Evaluating Text Generation with BERT
• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
• VL-BERT: Pre-training of Generic Visual-Linguistic Representations
• A Survey on Sentiment and Emotion Analysis for Computational Literary Studies
• The Data Science of Hollywood: Using Emotional Arcs of Movies to Drive Business Model Innovation in Entertainment Industries
• Beyond Narrative Description: Generating Poetry from Images by Multi-Adversarial Training