Script2Screen: Supporting Dialogue Scriptwriting with Interactive Audiovisual Generation

Abstract

Scriptwriting has traditionally been text-centric, a modality that only partially conveys the produced audiovisual experience. A formative study with professional writers informed us that connecting textual and audiovisual modalities can aid ideation and iteration, especially for writing dialogues. In this work, we present Script2Screen, an AI-assisted tool that integrates scriptwriting with audiovisual scene creation in a unified, synchronized workflow. Focusing on dialogues in scripts, Script2Screen generates expressive scenes with emotional speeches and animated characters through a novel text-to-audiovisual-scene pipeline. The user interface provides fine-grained controls, allowing writers to fine-tune audiovisual elements such as character gestures, speech emotions, and camera angles. A user study with both novice and professional writers from various domains demonstrated that Script2Screen's interactive audiovisual generation enhances the scriptwriting process, facilitating iterative refinement while complementing, rather than replacing, their creative efforts.

References in corpus (11)

• CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities
• Co-Writing with Opinionated Language Models Affects Users' Views
• Rhythmic Gesticulator: Rhythm-Aware Co-Speech Gesture Synthesis with Hierarchical Neural Embeddings
• RePrompt: Automatic Prompt Editing to Refine AI-Generative Art Towards Precise Expressions
• PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinement
• ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Models
• Rambler: Supporting Writing With Speech via LLM-Assisted Gist Manipulation
• Soloist: Generating Mixed-Initiative Tutorials from Existing Guitar Instructional Videos Through Audio Processing
• NewsPod: Automatic and Interactive News Podcasts
• Heteroglossia: In-Situ Story Ideation with the Crowd
• TaleStream: Supporting Story Ideation with Trope Knowledge