Patchview: LLM-Powered Worldbuilding with Generative Dust and Magnet Visualization

Abstract

Large language models (LLMs) can help writers build story worlds by generating world elements, such as factions, characters, and locations. However, making sense of many generated elements can be overwhelming. Moreover, if the user wants to precisely control aspects of generated elements that are difficult to specify verbally, prompting alone may be insufficient. We introduce Patchview, a customizable LLM-powered system that visually aids worldbuilding by allowing users to interact with story concepts and elements through the physical metaphor of magnets and dust. Elements in Patchview are visually dragged closer to concepts with high relevance, facilitating sensemaking. The user can also steer the generation with verbally elusive concepts by indicating the desired position of the element between concepts. When the user disagrees with the LLM's visualization and generation, they can correct those by repositioning the element. These corrections can be used to align the LLM's future behaviors to the user's perception. With a user study, we show that Patchview supports the sensemaking of world elements and steering of element generation, facilitating exploration during the worldbuilding process. Patchview provides insights on how customizable visual representation can help sensemake, steer, and align generative AI model behaviors with the user's intentions.

arXiv commentAccepted to UIST2024

References in corpus (14)

• Survey of Hallucination in Natural Language Generation
• Training language models to follow instructions with human feedback
• CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities
• Homogenization Effects of Large Language Models on Human Creative Ideation
• Sensecape: Enabling Multilevel Exploration and Sensemaking with Large Language Models
• ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing
• Graphologue: Exploring Large Language Model Responses with Interactive Diagrams
• A Design Space for Intelligent and Interactive Writing Assistants
• PromptPaint: Steering Text-to-Image Generation Through Paint Medium-like Interactions
• Spellburst: A Node-based Interface for Exploratory Creative Coding with Natural Language Prompts
• CALYPSO: LLMs as Dungeon Masters' Assistants
• Authors' Values and Attitudes Towards AI-bridged Scalable Personalization of Creative Language Arts
• Portrayal: Leveraging NLP and Visualization for Analyzing Fictional Characters
• TaleStream: Supporting Story Ideation with Trope Knowledge

Cited by in corpus (7)

• Creative Writers' Attitudes on Writing as Training Data for Large Language Models
• Toyteller: AI-powered Visual Storytelling Through Toy-Playing with Character Symbols
• Exploring Mobile Touch Interaction with Large Language Models
• DeepVIS: Bridging Natural Language and Data Visualization Through Step-wise Reasoning
• Steering AI-Driven Personalization of Scientific Text for General Audiences
• Spatial Balancing: Designing an LLM-Powered Spatial Externalization Interface for Iterative Science Communication Writing
• Constella: Supporting Storywriters' Interconnected Character Creation through LLM-based Multi-Agents