UXAgent: An LLM Agent-Based Usability Testing Framework for Web Design
arXiv:2502.12561 · doi:10.1145/3706599.3719729
Abstract
Usability testing is a fundamental yet challenging (e.g., inflexible to iterate the study design flaws and hard to recruit study participants) research method for user experience (UX) researchers to evaluate a web design. Recent advances in Large Language Model-simulated Agent (LLM-Agent) research inspired us to design UXAgent to support UX researchers in evaluating and reiterating their usability testing study design before they conduct the real human subject study. Our system features an LLM-Agent module and a universal browser connector module so that UX researchers can automatically generate thousands of simulated users to test the target website. The results are shown in qualitative (e.g., interviewing how an agent thinks ), quantitative (e.g., # of actions), and video recording formats for UX researchers to analyze. Through a heuristic user evaluation with five UX researchers, participants praised the innovation of our system but also expressed concerns about the future of LLM Agent-assisted UX study.
References in corpus (16)
- A Survey on Large Language Model based Autonomous Agents
- Usability testing: a review of some methodological and technical aspects of the method
- Human-AI Collaboration in Data Science: Exploring Data Scientists' Perceptions of Automated AI
- ChatDev: Communicative Agents for Software Development
- AXNav: Replaying Accessibility Tests from Natural Language
- Collaboration with Conversational AI Assistants for UX Evaluation: Questions and How to Ask them (Voice vs. Text)
- Agent Hospital: A Simulacrum of Hospital with Evolvable Medical Agents
- An Empathy-Based Sandbox Approach to Bridge the Privacy Gap among Attitudes, Goals, Knowledge, and Behaviors
- "Merging Results Is No Easy Task": An International Survey Study of Collaborative Data Analysis Practices Among UX Practitioners
- Agents for Automated User Experience Testing
- LLM-Augmented Agent-Based Modelling for Social Simulations: Challenges and Opportunities
- PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Action
- AgentOccam: A Simple Yet Strong Baseline for LLM-Based Web Agents
- Human-Centered Privacy Research in the Age of Large Language Models
- Privacy Leakage Overshadowed by Views of AI: A Study on Human Oversight of Privacy in Language Model Agent
- LASER: LLM Agent with State-Space Exploration for Web Navigation