Chatbots for Data Collection in Surveys: A Comparison of Four Theory-Based Interview Probes
arXiv:2503.08582 · doi:10.1145/3706598.3714128
Abstract
Surveys are a widespread method for collecting data at scale, but their rigid structure often limits the depth of qualitative insights obtained. While interviews naturally yield richer responses, they are challenging to conduct across diverse locations and large participant pools. To partially bridge this gap, we investigate the potential of using LLM-based chatbots to support qualitative data collection through interview probes embedded in surveys. We assess four theory-based interview probes: descriptive, idiographic, clarifying, and explanatory. Through a split-plot study design (N=64), we compare the probes' impact on response quality and user experience across three key stages of HCI research: exploration, requirements gathering, and evaluation. Our results show that probes facilitate the collection of high-quality survey data, with specific probes proving effective at different research stages. We contribute practical and methodological implications for using chatbots as research tools to enrich qualitative data collection.
CHI Conference on Human Factors in Computing Systems (CHI '25), April 26-May 1, 2025, Yokohama,Japan
References in corpus (6)
- Tell Me About Yourself: Using an AI-Powered Chatbot to Conduct Conversational Surveys with Open-ended Questions
- If I Hear You Correctly: Building and Evaluating Interview Chatbots with Active Listening Skills
- Calendar.help: Designing a Workflow-Based Scheduling Agent with Humans in the Loop
- Submitting surveys via a conversational interface: an evaluation of user acceptance and approach effectiveness
- Directed Diversity: Leveraging Language Embedding Distances for Collective Creativity in Crowd Ideation
- EmoWear: Exploring Emotional Teasers for Voice Message Interaction on Smartwatches