Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Data
arXiv:2301.05843 · doi:10.1145/3637364
Abstract
Large language models (LLMs) provide a new way to build chatbots by accepting natural language prompts. Yet, it is unclear how to design prompts to power chatbots to carry on naturalistic conversations while pursuing a given goal, such as collecting self-report data from users. We explore what design factors of prompts can help steer chatbots to talk naturally and collect data reliably. To this aim, we formulated four prompt designs with different structures and personas. Through an online study (N = 48) where participants conversed with chatbots driven by different designs of prompts, we assessed how prompt designs and conversation topics affected the conversation flows and users' perceptions of chatbots. Our chatbots covered 79% of the desired information slots during conversations, and the designs of prompts and topics significantly influenced the conversation flows and the data collection performance. We discuss the opportunities and challenges of building chatbots with LLMs.
arXiv comment22 pages including Appendix, 7 figures, 7 tables. Accepted to PACM HCI (CSCW 2024)
References in corpus (14)
- PaLM: Scaling Language Modeling with Pathways
- Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing
- CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities
- The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Development
- Neural Text Generation with Unlikelihood Training
- Tell Me About Yourself: Using an AI-Powered Chatbot to Conduct Conversational Surveys with Open-ended Questions
- What Makes Good In-Context Examples for GPT-?
- Holistic Evaluation of Language Models
- If I Hear You Correctly: Building and Evaluating Interview Chatbots with Active Listening Skills
- Effects of Persuasive Dialogues: Testing Bot Identities and Inquiry Strategies
- Submitting surveys via a conversational interface: an evaluation of user acceptance and approach effectiveness
- Developing a Personality Model for Speech-based Conversational Agents Using the Psycholexical Approach
- MyMove: Facilitating Older Adults to Collect In-Situ Activity Labels on a Smartwatch with Speech
- An Evaluation of Generative Pre-Training Model-based Therapy Chatbot for Caregivers
Cited by in corpus (10)
- A Systematic Review of Generative AI for Teaching and Learning Practice
- MindfulDiary: Harnessing Large Language Model to Support Psychiatric Patients' Journaling
- Understanding the Impact of Long-Term Memory on Self-Disclosure with Large Language Model-Driven Chatbots for Public Health Intervention
- ChaCha: Leveraging Large Language Models to Prompt Children to Share Their Emotions about Personal Events
- MindShift: Leveraging Large Language Models for Mental-States-Based Problematic Smartphone Use Intervention
- ExploreSelf: Fostering User-driven Exploration and Reflection on Personal Challenges with Adaptive Guidance by Large Language Models
- Reviving Static Charts into Live Charts
- Comparing Large Language Model AI and Human-Generated Coaching Messages for Behavioral Weight Loss
- AI-Enabled Conversational Journaling for Advancing Parkinson's Disease Symptom Tracking
- RemVerse: Supporting Reminiscence Activities for Older Adults through AI-Assisted Virtual Reality