Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers

Abstract

Should a large language model (LLM) be used as a therapist? In this paper, we investigate the use of LLMs to *replace* mental health providers, a use case promoted in the tech startup and research space. We conduct a mapping review of therapy guides used by major medical institutions to identify crucial aspects of therapeutic relationships, such as the importance of a therapeutic alliance between therapist and client. We then assess the ability of LLMs to reproduce and adhere to these aspects of therapeutic relationships by conducting several experiments investigating the responses of current LLMs, such as `gpt-4o`. Contrary to best practices in the medical community, LLMs 1) express stigma toward those with mental health conditions and 2) respond inappropriately to certain common (and critical) conditions in naturalistic therapy settings -- e.g., LLMs encourage clients' delusional thinking, likely due to their sycophancy. This occurs even with larger and newer LLMs, indicating that current safety practices may not address these gaps. Furthermore, we note foundational and practical barriers to the adoption of LLMs as therapists, such as that a therapeutic alliance requires human characteristics (e.g., identity and stakes). For these reasons, we conclude that LLMs should not replace therapists, and we discuss alternative roles for LLMs in clinical therapy.

References in corpus (35)

• Intelligent Clinical Documentation: Harnessing Generative AI for Patient-Centric Clinical Note Generation
• Roles for Computing in Social Change
• Towards Understanding Sycophancy in Language Models
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• ChatCounselor: A Large Language Models for Mental Health Support
• Psy-LLM: Scaling up Global Mental Health Psychological Services with AI-based Large Language Models
• Beyond Preferences in AI Alignment
• A Computational Framework for Behavioral Assessment of LLM Therapists
• Exploring The Design of Prompts For Applying GPT-3 based Chatbots: A Mental Wellbeing Case Study on Mechanical Turk
• Understanding Social Reasoning in Language Models with Language Models
• Creating Trustworthy LLMs: Dealing with Hallucinations in Healthcare AI
• The Typing Cure: Experiences with Large Language Model Chatbots for Mental Health Support
• SafetyBench: Evaluating the Safety of Large Language Models
• Benefits and Harms of Large Language Models in Digital Mental Health
• LLM-Powered Socially Assistive Robot-Delivered Cognitive Behavioral Therapy Exercises: an Exploratory Study with University Students
• Addressing cognitive bias in medical language models
• Evaluating and Mitigating Discrimination in Language Model Decisions
• Bias patterns in the application of LLMs for clinical decision support: A comprehensive study
• Response Generation for Cognitive Behavioral Therapy with Large Language Models: Comparative Study with Socratic Questioning
• Does Refusal Training in LLMs Generalize to the Past Tense?
• Risks from Language Models for Automated Mental Healthcare: Ethics and Structure for Implementation
• Roleplay-doh: Enabling Domain-Experts to Create LLM-simulated Patients via Eliciting and Adhering to Principles
• Evaluating the Efficacy of Interactive Language Therapy Based on LLM for High-Functioning Autistic Adolescent Psychological Counseling
• Therapy as an NLP Task: Comparing LLMs and Human Peers Behaviors in CBT Sessions
• A Survey of Large Language Models in Psychotherapy: Current Landscape and Future Directions
• Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?
• Can Large Language Models Identify Authorship?
• Language Models as Critical Thinking Tools: A Case Study of Philosophers
• Do Large Language Models Align with Core Mental Health Counseling Competencies?
• CBT-Bench: Evaluating Large Language Models on Assisting Cognitive Behavior Therapy
• SimpleToM: Exposing the Gap between Explicit ToM Inference and Implicit ToM Application in LLMs
• Are Large Language Models Consistent over Value-laden Questions?
• Inducing Positive Perspectives with Text Reframing
• Enhancing Psychotherapy Counseling: A Data Augmentation Pipeline Leveraging Large Language Models for Counseling Conversations
• MentalArena: Self-play Training of Language Models for Diagnosis and Treatment of Mental Health Disorders

Cited by in corpus (4)

• ELEPHANT: Measuring and understanding social sycophancy in LLMs
• AI Chatbot Suicide Risk Detection and Response: Human Validation Study of the Open-Source VERA-MH Safety Evaluation
• Chaplains' Reflections on the Design and Usage of AI for Conversational Care
• When Vocabulary Comprehension Fails Clinical Reasoning: Evaluating Therapy Bots' Safety Risks for Generation Alpha