"It's a Fair Game", or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agents

Abstract

The widespread use of Large Language Model (LLM)-based conversational agents (CAs), especially in high-stakes domains, raises many privacy concerns. Building ethical LLM-based CAs that respect user privacy requires an in-depth understanding of the privacy risks that concern users the most. However, existing research, primarily model-centered, does not provide insight into users' perspectives. To bridge this gap, we analyzed sensitive disclosures in real-world ChatGPT conversations and conducted semi-structured interviews with 19 LLM-based CA users. We found that users are constantly faced with trade-offs between privacy, utility, and convenience when using LLM-based CAs. However, users' erroneous mental models and the dark patterns in system design limited their awareness and comprehension of the privacy risks. Additionally, the human-like interactions encouraged more sensitive disclosures, which complicated users' ability to navigate the trade-offs. We discuss practical design guidelines and the needs for paradigm shifts to protect the privacy of LLM-based CA users.

26 pages, 5 figures

References in corpus (2)

• Training language models to follow instructions with human feedback
• Extracting Training Data from Large Language Models

Cited by in corpus (13)

• Evaluation and Benchmarking of LLM Agents: A Survey
• Embedding Large Language Models into Extended Reality: Opportunities and Challenges for Inclusion, Engagement, and Privacy
• Using AI Assistants in Software Development: A Qualitative Study on Security Practices and Concerns
• "I've talked to ChatGPT about my issues last night.": Examining Mental Health Conversations with Large Language Models through Reddit Analysis
• Secret Use of Large Language Model (LLM)
• Rescriber: Smaller-LLM-Powered User-Led Data Minimization for LLM-Based Chatbots
• Understanding Users' Security and Privacy Concerns and Attitudes Towards Conversational AI Platforms
• Objection Overruled! Lay People can Distinguish Large Language Models from Lawyers, but still Favour Advice from an LLM
• Cracking Aegis: An Adversarial LLM-based Game for Raising Awareness of Vulnerabilities in Privacy Protection
• The Emerged Security and Privacy of LLM Agent: A Survey with Case Studies
• The Impact of a Chatbot's Ephemerality-Framing on Self-Disclosure Perceptions
• NoRe: Augmenting Journaling Experience with Generative AI for Music Creation
• One Size Fits All? A Modular Adaptive Sanitization Kit (MASK) for Customizable Privacy-Preserving Phone Scam Detection