Generative Language Models and Automated Influence Operations: Emerging Threats and Potential Mitigations

Abstract

Generative language models have improved drastically, and can now produce realistic text outputs that are difficult to distinguish from human-written content. For malicious actors, these language models bring the promise of automating the creation of convincing and misleading text for use in influence operations. This report assesses how language models might change influence operations in the future, and what steps can be taken to mitigate this threat. We lay out possible changes to the actors, behaviors, and content of online influence operations, and provide a framework for stages of the language model-to-influence operations pipeline that mitigations could target (model construction, model access, content dissemination, and belief formation). While no reasonable mitigation can be expected to fully prevent the threat of AI-enabled influence operations, a combination of multiple mitigations may make an important difference.

arXiv comment82 pages, 26 figures

Cited by in corpus (10)

• Human heuristics for AI-generated language are flawed
• Anatomy of an AI-powered malicious social botnet
• Ethics of generative AI and manipulation: a design-oriented research agenda
• The AI Community Building the Future? A Quantitative Analysis of Development Activity on Hugging Face Hub
• Ultrafast Coherent Dynamics of Microring Modulators
• A Survey on Automatic Credibility Assessment Using Textual Credibility Signals in the Era of Large Language Models
• Cross-Attention Watermarking of Large Language Models
• "There Has To Be a Lot That We're Missing": Moderating AI-Generated Content on Reddit
• ChatGPT for President! Presupposed content in politicians versus GPT-generated texts
• Information Retrieval in the Age of Generative AI: The RGB Model