Multimodal Story Generation on Plural Images

Abstract

Traditionally, text generation models take in a sequence of text as input, and iteratively generate the next most probable word using pre-trained parameters. In this work, we propose the architecture to use images instead of text as the input of the text generation model, called StoryGen. In the architecture, we design a Relational Text Data Generator algorithm that relates different features from multiple images. The output samples from the model demonstrate the ability to generate meaningful paragraphs of text containing the extracted features from the input images. This is an undergraduate project report. Completed Dec. 2019 at the Cooper Union.

This is an undergraduate project report. Completed Dec. 2019 at the Cooper Union

References in corpus (5)

• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
• Semi-supervised Sequence Learning
• Long Text Generation via Adversarial Training with Leaked Information
• Deep Visual-Semantic Alignments for Generating Image Descriptions