AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathways

Abstract

An Artificial Intelligence (AI) agent is a software entity that autonomously performs tasks or makes decisions based on pre-defined objectives and data inputs. AI agents, capable of perceiving user inputs, reasoning and planning tasks, and executing actions, have seen remarkable advancements in algorithm development and task performance. However, the security challenges they pose remain under-explored and unresolved. This survey delves into the emerging security threats faced by AI agents, categorizing them into four critical knowledge gaps: unpredictability of multi-step user inputs, complexity in internal executions, variability of operational environments, and interactions with untrusted external entities. By systematically reviewing these threats, this paper highlights both the progress made and the existing limitations in safeguarding AI agents. The insights provided aim to inspire further research into addressing the security threats associated with AI agents, thereby fostering the development of more robust and secure AI agent applications.

arXiv commentSubmitted to ACM Computing Survey

References in corpus (19)

• Survey of Hallucination in Natural Language Generation
• Training language models to follow instructions with human feedback
• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly
• Self-Consistency Improves Chain of Thought Reasoning in Language Models
• Tree of Thoughts: Deliberate Problem Solving with Large Language Models
• Deep reinforcement learning from human preferences
• Toolformer: Language Models Can Teach Themselves to Use Tools
• A Survey on ChatGPT: AI-Generated Contents, Challenges, and Solutions
• HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
• Februus: Input Purification Defense Against Trojan Attacks on Deep Neural Network Systems
• MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots
• Factuality Enhanced Language Models for Open-Ended Text Generation
• From ChatGPT to ThreatGPT: Impact of Generative AI in Cybersecurity and Privacy
• FuzzLLM: A Novel and Universal Fuzzing Framework for Proactively Discovering Jailbreak Vulnerabilities in Large Language Models
• MarioGPT: Open-Ended Text2Level Generation through Large Language Models
• Toxicity Detection with Generative Prompt-based Inference
• Efficient Adversarial Attacks on Online Multi-agent Reinforcement Learning
• Robust Multi-Agent Reinforcement Learning via Adversarial Regularization: Theoretical Foundation and Stable Algorithms

Cited by in corpus (5)

• AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges
• CyberRAG: An Agentic RAG cyber attack classification and reporting tool
• A Survey on Agent Workflow -- Status and Future
• From Prompt Injections to Protocol Exploits: Threats in LLM-Powered AI Agents Workflows
• From bits to atoms: A survey of cross-layer safety in Embodied AI