Informative Visual Storytelling with Cross-modal Rules

Abstract

Existing methods in the Visual Storytelling field often suffer from the problem of generating general descriptions, while the image contains a lot of meaningful contents remaining unnoticed. The failure of informative story generation can be concluded to the model's incompetence of capturing enough meaningful concepts. The categories of these concepts include entities, attributes, actions, and events, which are in some cases crucial to grounded storytelling. To solve this problem, we propose a method to mine the cross-modal rules to help the model infer these informative concepts given certain visual input. We first build the multimodal transactions by concatenating the CNN activations and the word indices. Then we use the association rule mining algorithm to mine the cross-modal rules, which will be used for the concept inference. With the help of the cross-modal rules, the generated stories are more grounded and informative. Besides, our proposed method holds the advantages of interpretation, expandability, and transferability, indicating potential for wider application. Finally, we leverage these concepts in our encoder-decoder framework with the attention mechanism. We conduct several experiments on the VIsual StoryTelling~(VIST) dataset, the results of which demonstrate the effectiveness of our approach in terms of both automatic metrics and human evaluation. Additional experiments are also conducted showing that our mined cross-modal rules as additional knowledge helps the model gain better performance when trained on a small dataset.

arXiv comment9 pages, to appear in ACM Multimedia 2019

References in corpus (8)

• Neural Machine Translation by Jointly Learning to Align and Translate
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• GLAC Net: GLocal Attention Cascading Networks for Multi-image Cued Story Generation
• Hierarchically Structured Reinforcement Learning for Topically Coherent Visual Story Generation
• Using Inter-Sentence Diverse Beam Search to Reduce Redundancy in Visual Storytelling
• Rare Association Rule Mining for Network Intrusion Detection
• Anomaly Detection in XML-Structured SOAP Messages Using Tree-Based Association Rule Mining

Cited by in corpus (4)

• A Survey of Natural Language Generation
• Topic Adaptation and Prototype Encoding for Few-Shot Visual Storytelling
• Multi-Modal Experience Inspired AI Creation
• BERT-hLSTMs: BERT and Hierarchical LSTMs for Visual Storytelling