Exploring LLMs for Automated Generation and Adaptation of Questionnaires

Abstract

Effective questionnaire design improves the validity of the results, but creating and adapting questionnaires across contexts is challenging due to resource constraints and limited expert access. Recently, the emergence of LLMs has led researchers to explore their potential in survey research. In this work, we focus on the suitability of LLMs in assisting the generation and adaptation of questionnaires. We introduce a novel pipeline that leverages LLMs to create new questionnaires, pretest with a target audience to determine potential issues and adapt existing standardized questionnaires for different contexts. We evaluated our pipeline for creation and adaptation through two studies on Prolific, involving 238 participants from the US and 118 participants from South Africa. Our findings show that participants found LLM-generated text clearer, LLM-pretested text more specific, and LLM-adapted questions slightly clearer and less biased than traditional ones. Our work opens new opportunities for LLM-driven questionnaire support in survey research.

Published in the Proceedings of the 7th ACM Conference on Conversational User Interfaces (CUI '25)

References in corpus (13)

• Out of One, Many: Using Language Models to Simulate Human Samples
• Unmasking and Quantifying Racial Bias of Large Language Models in Medical Report Generation
• Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus?
• ERNIE-GEN: An Enhanced Multi-Flow Pre-training and Fine-tuning Framework for Natural Language Generation
• Large language models that replace human participants can harmfully misportray and flatten identity groups
• Questioning the Survey Responses of Large Language Models
• Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs
• What should I Ask: A Knowledge-driven Approach for Follow-up Questions Generation in Conversational Surveys
• Do LLMs exhibit human-like response biases? A case study in survey design
• Collecting Qualitative Data at Scale with Large Language Models: A Case Study
• Questionnaires for Everyone: Streamlining Cross-Cultural Questionnaire Adaptation with GPT-Based Translation Quality Evaluation
• ChatGPTest: opportunities and cautionary tales of utilizing AI for questionnaire pretesting
• $SmartProbe$: A Virtual Moderator for Market Research Surveys