Tell Me About Yourself: Using an AI-Powered Chatbot to Conduct Conversational Surveys with Open-ended Questions

Abstract

The rise of increasingly more powerful chatbots offers a new way to collect information through conversational surveys, where a chatbot asks open-ended questions, interprets a user's free-text responses, and probes answers whenever needed. To investigate the effectiveness and limitations of such a chatbot in conducting surveys, we conducted a field study involving about 600 participants. In this study with mostly open-ended questions, half of the participants took a typical online survey on Qualtrics and the other half interacted with an AI-powered chatbot to complete a conversational survey. Our detailed analysis of over 5200 free-text responses revealed that the chatbot drove a significantly higher level of participant engagement and elicited significantly better quality responses measured by Gricean Maxims in terms of their informativeness, relevance, specificity, and clarity. Based on our results, we discuss design implications for creating AI-powered chatbots to conduct effective surveys and beyond.

The paper is accepted by ACM Transactions on Computer-Human Interaction (TOCHI)

References in corpus (3)

• PARADISE: A Framework for Evaluating Spoken Dialogue Agents
• Calendar.help: Designing a Workflow-Based Scheduling Agent with Humans in the Loop
• Confiding in and Listening to Virtual Agents: The Effect of Personality

Cited by in corpus (14)

• UX Research on Conversational Human-AI Interaction: A Literature Review of the ACM Digital Library
• Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Data
• Designing Effective Interview Chatbots: Automatic Chatbot Profiling and Design Suggestion Generation for Chatbot Debugging
• Inform the uninformed: Improving Online Informed Consent Reading with an AI-Powered Chatbot
• User Interaction Patterns and Breakdowns in Conversing with LLM-Powered Voice Assistants
• Chatbots for Data Collection in Surveys: A Comparison of Four Theory-Based Interview Probes
• Comparing How a Chatbot References User Utterances from Previous Chatting Sessions: An Investigation of Users' Privacy Concerns and Perceptions
• On LLM Wizards: Identifying Large Language Models' Behaviors for Wizard of Oz Experiments
• AINeedsPlanner: A Workbook to Support Effective Collaboration Between AI Experts and Clients
• Democratizing Chatbot Debugging: A Computational Framework for Evaluating and Explaining Inappropriate Chatbot Responses
• VizTrust: A Visual Analytics Tool for Capturing User Trust Dynamics in Human-AI Communication
• "Mango Mango, How to Let The Lettuce Dry Without A Spinner?": Exploring User Perceptions of Using An LLM-Based Conversational Assistant Toward Cooking Partner
• Card Sorting with Fewer Cards and the Same Mental Models? A Re-examination of an Established Practice
• Dynamic Surveys: Using LLMs to Blend Qualitative Depth,Quantitative Structure, and Collaborative Interaction