Anatomy of an AI-powered malicious social botnet

Abstract

Large language models (LLMs) exhibit impressive capabilities in generating realistic text across diverse subjects. Concerns have been raised that they could be utilized to produce fake content with a deceptive intention, although evidence thus far remains anecdotal. This paper presents a case study about a Twitter botnet that appears to employ ChatGPT to generate human-like content. Through heuristics, we identify 1,140 accounts and validate them via manual annotation. These accounts form a dense cluster of fake personas that exhibit similar behaviors, including posting machine-generated content and stolen images, and engage with each other through replies and retweets. ChatGPT-generated content promotes suspicious websites and spreads harmful comments. While the accounts in the AI botnet can be detected through their coordination patterns, current state-of-the-art LLM content classifiers fail to discriminate between them and human accounts in the wild. These findings highlight the threats posed by AI-enabled social bots.

References in corpus (27)

• Training language models to follow instructions with human feedback
• On the Opportunities and Risks of Foundation Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• Arming the public with artificial intelligence to counter social bots
• Human heuristics for AI-generated language are flawed
• How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection
• AI model GPT-3 (dis)informs us better than humans
• Voyager: An Open-Ended Embodied Agent with Large Language Models
• A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Models
• Can AI-Generated Text be Reliably Detected?
• Botometer 101: Social bot practicum for computational social scientists
• DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature
• Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond
• Generative Language Models and Automated Influence Operations: Emerging Threats and Potential Mitigations
• A Watermark for Large Language Models
• CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society
• Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense
• On the Possibilities of AI-Generated Text Detection
• The Science of Detecting LLM-Generated Texts
• ChatGPT: Applications, Opportunities, and Threats
• GPT detectors are biased against non-native English writers
• Spear Phishing With Large Language Models
• Fundamentals of Generative Large Language Models and Perspectives in Cyber-Defense
• Stylometric Detection of AI-Generated Text in Twitter Timelines
• Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methods
• Protecting Language Generation Models via Invisible Watermarking
• Machine-Made Media: Monitoring the Mobilization of Machine-Generated Articles on Misinformation and Mainstream News Websites

Cited by in corpus (9)

• GenAI Against Humanity: Nefarious Applications of Generative Artificial Intelligence and Large Language Models
• Factuality Challenges in the Era of Large Language Models
• A new sociology of humans and machines
• Measure-Observe-Remeasure: An Interactive Paradigm for Differentially-Private Exploratory Analysis
• Fact-checking information from large language models can decrease headline discernment
• AI Rules? Characterizing Reddit Community Policies Towards AI-Generated Content
• "There Has To Be a Lot That We're Missing": Moderating AI-Generated Content on Reddit
• Beyond Binary: Towards Fine-Grained LLM-Generated Text Detection via Role Recognition and Involvement Measurement
• Deterrence Effects of Social Media Interventions on Health Misinformation Dissemination by Bots and Humans