Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistant
arXiv:2502.01390 · doi:10.1145/3706598.3713218
Abstract
Since the explosion in popularity of ChatGPT, large language models (LLMs) have continued to impact our everyday lives. Equipped with external tools that are designed for a specific purpose (e.g., for flight booking or an alarm clock), LLM agents exercise an increasing capability to assist humans in their daily work. Although LLM agents have shown a promising blueprint as daily assistants, there is a limited understanding of how they can provide daily assistance based on planning and sequential decision making capabilities. We draw inspiration from recent work that has highlighted the value of 'LLM-modulo' setups in conjunction with humans-in-the-loop for planning tasks. We conducted an empirical study (N = 248) of LLM agents as daily assistants in six commonly occurring tasks with different levels of risk typically associated with them (e.g., flight ticket booking and credit card payments). To ensure user agency and control over the LLM agent, we adopted LLM agents in a plan-then-execute manner, wherein the agents conducted step-wise planning and step-by-step execution in a simulation environment. We analyzed how user involvement at each stage affects their trust and collaborative team performance. Our findings demonstrate that LLM agents can be a double-edged sword -- (1) they can work well when a high-quality plan and necessary user involvement in execution are available, and (2) users can easily mistrust the LLM agents with plans that seem plausible. We synthesized key insights for using LLM agents as daily assistants to calibrate user trust and achieve better overall task outcomes. Our work has important implications for the future design of daily assistants and human-AI collaboration with LLM agents.
conditionally accepted to CHI 2025
References in corpus (15)
- Survey of Hallucination in Natural Language Generation
- A Survey on Large Language Model based Autonomous Agents
- To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making
- Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making
- Autonomous Agents Modelling Other Agents: A Comprehensive Survey and Open Problems
- Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations
- Designing for Responsible Trust in AI Systems: A Communication Perspective
- "I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust
- Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Making
- Knowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI Systems
- Human-AI Collaboration: The Effect of AI Delegation on Human Task Performance and Task Satisfaction
- Human Perceptions on Moral Responsibility of AI: A Case Study in AI-Assisted Bail Decision-Making
- Algorithmic Risk Assessments Can Alter Human Decision-Making Processes in High-Stakes Government Contexts
- Is Conversational XAI All You Need? Human-AI Decision Making With a Conversational XAI Assistant
- If in a Crowdsourced Data Annotation Pipeline, a GPT-4