FairyTailor: A Multimodal Generative Framework for Storytelling

Abstract

Storytelling is an open-ended task that entails creative thinking and requires a constant flow of ideas. Natural language generation (NLG) for storytelling is especially challenging because it requires the generated text to follow an overall theme while remaining creative and diverse to engage the reader. In this work, we introduce a system and a web-based demo, FairyTailor, for human-in-the-loop visual story co-creation. Users can create a cohesive children's fairytale by weaving generated texts and retrieved images with their input. FairyTailor adds another modality and modifies the text generation process to produce a coherent and creative sequence of text and images. To our knowledge, this is the first dynamic tool for multimodal story generation that allows interactive co-formation of both texts and images. It allows users to give feedback on co-created stories and share their results.

arXiv commentvisit https://fairytailor.org/ and https://github.com/EdenBD/MultiModalStory-demo for web demo and source code

References in corpus (8)

• Learning Transferable Visual Models From Natural Language Supervision
• Deep Residual Learning for Image Recognition
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Language Models are Few-Shot Learners
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Story Generation from Sequence of Independent Short Descriptions
• Narrative Interpolation for Generating and Understanding Stories

Cited by in corpus (2)

• Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora
• Visual Story-Writing: Writing by Manipulating Visual Representations of Stories