The Dark Side of AI Companionship: A Taxonomy of Harmful Algorithmic Behaviors in Human-AI Relationships

Abstract

As conversational AI systems increasingly permeate the socio-emotional realms of human life, they bring both benefits and risks to individuals and society. Despite extensive research on detecting and categorizing harms in AI systems, less is known about the harms that arise from social interactions with AI chatbots. Through a mixed-methods analysis of 35,390 conversation excerpts shared on r/replika, an online community for users of the AI companion Replika, we identified six categories of harmful behaviors exhibited by the chatbot: relational transgression, verbal abuse and hate, self-inflicted harm, harassment and violence, mis/disinformation, and privacy violations. The AI contributes to these harms through four distinct roles: perpetrator, instigator, facilitator, and enabler. Our findings highlight the relational harms of AI chatbots and the danger of algorithmic compliance, enhancing the understanding of AI harms in socio-emotional interactions. We also provide suggestions for designing ethical and responsible AI systems that prioritize user safety and well-being.

References in corpus (12)

• ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks
• The Fallacy of AI Functionality
• Co-Writing with Opinionated Language Models Affects Users' Views
• Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech
• Harms from Increasingly Agentic Algorithmic Systems
• A Framework of Severity for Harmful Content Online
• Trauma-Informed Social Media: Towards Solutions for Reducing and Healing Online Harm
• Understanding the Impact of Long-Term Memory on Self-Disclosure with Large Language Model-Driven Chatbots for Public Health Intervention
• Not My Voice! A Taxonomy of Ethical and Safety Harms of Speech Generators
• Beyond Preferences in AI Alignment
• Blaming Humans and Machines: What Shapes People's Reactions to Algorithmic Harm
• Participation and Division of Labor in User-Driven Algorithm Audits: How Do Everyday Users Work together to Surface Algorithmic Harms?

Cited by in corpus (5)

• Measuring Machine Companionship: Scale Development and Validation for AI Companions
• Restoration, Exploration and Transformation: How Youth Engage Character.AI Chatbots for Feels, Fun and Finding themselves
• Chaplains' Reflections on the Design and Usage of AI for Conversational Care
• A Risk Ontology for Evaluating AI-Powered Psychotherapy Virtual Agents
• Fast-Food Intimacy: How Chinese Women Navigate Soul's AI Boyfriend