The Impact of Multiple Parallel Phrase Suggestions on Email Input and Composition Behaviour of Native and Non-Native English Writers

Abstract

We present an in-depth analysis of the impact of multi-word suggestion choices from a neural language model on user behaviour regarding input and text composition in email writing. Our study for the first time compares different numbers of parallel suggestions, and use by native and non-native English writers, to explore a trade-off of "efficiency vs ideation", emerging from recent literature. We built a text editor prototype with a neural language model (GPT-2), refined in a prestudy with 30 people. In an online study (N=156), people composed emails in four conditions (0/1/3/6 parallel suggestions). Our results reveal (1) benefits for ideation, and costs for efficiency, when suggesting multiple phrases; (2) that non-native speakers benefit more from more suggestions; and (3) further insights into behaviour patterns. We discuss implications for research, the design of interactive suggestion systems, and the vision of supporting writers with AI instead of replacing them.

arXiv comment21 pages, 4 figures, ACM CHI 2021

References in corpus (9)

• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Language Models are Few-Shot Learners
• Model Cards for Model Reporting
• Pre-trained Models for Natural Language Processing: A Survey
• Release Strategies and the Social Impacts of Language Models
• Controllable Neural Story Plot Generation via Reward Shaping
• Conversations with Documents. An Exploration of Document-Centered Assistance
• Diversifying Reply Suggestions using a Matching-Conditional Variational Autoencoder
• Counterfactual Language Model Adaptation for Suggesting Phrases

Cited by in corpus (29)

• CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities
• Co-Writing with Opinionated Language Models Affects Users' Views
• Large-scale Text-to-Image Generation Models for Visual Artists' Creative Works
• The AI Ghostwriter Effect: When Users Do Not Perceive Ownership of AI-Generated Text But Self-Declare as Authors
• A Design Space for Intelligent and Interactive Writing Assistants
• Beyond Text Generation: Supporting Writers with Continuous Automatic Text Summaries
• Choice Over Control: How Users Write with Large Language Models using Diegetic and Non-Diegetic Prompting
• AI Suggestions Homogenize Writing Toward Western Styles and Diminish Cultural Nuances
• LaMPost: Design and Evaluation of an AI-assisted Email Writing Prototype for Adults with Dyslexia
• A Taxonomy for Human-LLM Interaction Modes: An Initial Exploration
• ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Models
• Interacting with next-phrase suggestions: How suggestion systems aid and influence the cognitive processes of writing
• Comparing Sentence-Level Suggestions to Message-Level Suggestions in AI-Mediated Communication
• Help Me Reflect: Leveraging Self-Reflection Interface Nudges to Enhance Deliberativeness on Online Deliberation Platforms
• Suggestion Lists vs. Continuous Generation: Interaction Design for Writing with Generative Models on Mobile Devices Affect Text Length, Wording and Perceived Authorship
• Storyfier: Exploring Vocabulary Learning Support with Text Generation Models
• Co-Writing with AI, on Human Terms: Aligning Research with User Demands Across the Writing Process
• Authors' Values and Attitudes Towards AI-bridged Scalable Personalization of Creative Language Arts
• Reviewriter: AI-Generated Instructions For Peer Review Writing
• Collage is the New Writing: Exploring the Fragmentation of Text and User Interfaces in AI Tools
• LLM-Mediated Domain-Specific Voice Agents: The Case of TextileBot
• VideoDiff: Human-AI Video Co-Creation with Alternatives
• SynthBio: A Case Study in Human-AI Collaborative Curation of Text Datasets
• CoCo Matrix: Taxonomy of Cognitive Contributions in Co-writing with Intelligent Agents
• Composable Prompting Workspaces for Creative Writing: Exploration and Iteration Using Dynamic Widgets
• Content-Driven Local Response: Supporting Sentence-Level and Message-Level Mobile Email Replies With and Without AI
• Visual Story-Writing: Writing by Manipulating Visual Representations of Stories
• StudyAlign: A Software System for Conducting Web-Based User Studies with Functional Interactive Prototypes
• Inspiration through Observation: Demonstrating the Influence of Automatically Generated Text on Creative Writing