A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly

Abstract

Large Language Models (LLMs), such as ChatGPT and Bard, have revolutionized natural language understanding and generation. They possess deep language comprehension, human-like text generation capabilities, contextual awareness, and robust problem-solving skills, making them invaluable in various domains (e.g., search engines, customer support, translation). In the meantime, LLMs have also gained traction in the security community, revealing security vulnerabilities and showcasing their potential in security-related tasks. This paper explores the intersection of LLMs with security and privacy. Specifically, we investigate how LLMs positively impact security and privacy, potential risks and threats associated with their use, and inherent vulnerabilities within LLMs. Through a comprehensive literature review, the paper categorizes the papers into "The Good" (beneficial LLM applications), "The Bad" (offensive applications), and "The Ugly" (vulnerabilities of LLMs and their defenses). We have some interesting findings. For example, LLMs have proven to enhance code security (code vulnerability detection) and data privacy (data confidentiality protection), outperforming traditional methods. However, they can also be harnessed for various attacks (particularly user-level attacks) due to their human-like reasoning abilities. We have identified areas that require further research efforts. For example, Research on model and parameter extraction attacks is limited and often theoretical, hindered by LLM parameter scale and confidentiality. Safe instruction tuning, a recent development, requires more exploration. We hope that our work can shed light on the LLMs' potential to both bolster and jeopardize cybersecurity.

References in corpus (109)

• Training language models to follow instructions with human feedback
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• A Survey of Large Language Models
• Gender bias and stereotypes in Large Language Models
• BloombergGPT: A Large Language Model for Finance
• CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data
• Systematic Classification of Side-Channel Attacks: A Case Study for Mobile Devices
• Universal and Transferable Adversarial Attacks on Aligned Language Models
• Language Models (Mostly) Know What They Know
• The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only
• Large Language Models for Code: Security Hardening and Adversarial Testing
• LIMA: Less Is More for Alignment
• Getting pwn'd by AI: Penetration Testing with Large Language Models
• MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots
• Eight Things to Know about Large Language Models
• Fake News in Sheep's Clothing: Robust Fake News Detection Against LLM-Empowered Style Attacks
• Adversarial Training for Large Neural Language Models
• Prompt Injection attack against LLM-integrated Applications
• Towards Demystifying Membership Inference Attacks
• Large Language Models Can Be Strong Differentially Private Learners
• Jailbroken: How Does LLM Safety Training Fail?
• LLM in the Shell: Generative Honeypots
• Decoding the Threat Landscape : ChatGPT, FraudGPT, and WormGPT in Social Engineering Attacks
• CodeT: Code Generation with Generated Tests
• The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset
• Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision
• LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examples
• Beyond the Safeguards: Exploring the Security Risks of ChatGPT
• No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generation
• Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs
• Physical Side-Channel Attacks on Embedded Neural Networks: A Survey
• PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts
• "Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models
• Prompt as Triggers for Backdoor Attack: Examining the Vulnerability in Language Models
• FuzzLLM: A Novel and Universal Fuzzing Framework for Proactively Discovering Jailbreak Vulnerabilities in Large Language Models
• Poisoning Language Models During Instruction Tuning
• Beyond Memorization: Violating Privacy Via Inference with Large Language Models
• Can LLM-Generated Misinformation Be Detected?
• Baseline Defenses for Adversarial Attacks Against Aligned Language Models
• Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks
• RRHF: Rank Responses to Align Language Models with Human Feedback without tears
• LLM Self Defense: By Self Examination, LLMs Know They Are Being Tricked
• FATE-LLM: A Industrial Grade Federated Learning Framework for Large Language Models
• SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks
• PentestGPT: An LLM-empowered Automatic Penetration Testing Tool
• How Should Pre-Trained Language Models Be Fine-Tuned Towards Adversarial Robustness?
• Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilities
• Conversational Health Agents: A Personalized LLM-Powered Agent Framework
• From Text to MITRE Techniques: Exploring the Malicious Use of Large Language Models for Generating Cyber Attack Payloads
• AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models
• Comparing Traditional and LLM-based Search for Consumer Choice: A Randomized Experiment
• ChatUniTest: A Framework for LLM-Based Test Generation
• Large Language Models are Edge-Case Fuzzers: Testing Deep Learning Libraries via FuzzGPT
• Evaluation of ChatGPT Model for Vulnerability Detection
• On the Reliability of Watermarks for Large Language Models
• Low-code LLM: Graphical User Interface over Large Language Models
• Detecting Phishing Sites Using ChatGPT
• Scaling Laws and Interpretability of Learning from Repeated Data
• Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations
• GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts
• Benchmarking Machine Translation with Cultural Awareness
• Chatbots to ChatGPT in a Cybersecurity Space: Evolution, Vulnerabilities, Attacks, Challenges, and Future Recommendations
• On The Impact of Machine Learning Randomness on Group Fairness
• Augmenting Greybox Fuzzing with Generative AI
• LLMs as Hackers: Autonomous Linux Privilege Escalation Attacks
• RatGPT: Turning online LLMs into Proxies for Malware Attacks
• How Prevalent is Gender Bias in ChatGPT? -- Exploring German and English ChatGPT Responses
• On the Exploitability of Instruction Tuning
• Membership Inference on Word Embedding and Beyond
• DIVAS: An LLM-based End-to-End Framework for SoC Security Analysis and Policy-based Protection
• Devising and Detecting Phishing: Large Language Models vs. Smaller Human Models
• Fake News Detectors are Biased against Texts Generated by Large Language Models
• Text Revealer: Private Text Reconstruction via Model Inversion Attacks against Transformers
• Nightshade: Prompt-Specific Poisoning Attacks on Text-to-Image Generative Models
• Binary Code Summarization: Benchmarking ChatGPT/GPT-4 and Other Large Language Models
• Pop Quiz! Can a Large Language Model Help With Reverse Engineering?
• Differentially Private Decoding in Large Language Models
• Adversarial Attacks and Defenses in Large Language Models: Old and New Threats
• Weakly Supervised Veracity Classification with LLM-Predicted Credibility Signals
• Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions
• AutoDroid: LLM-powered Task Automation in Android
• Defending Pre-trained Language Models as Few-shot Learners against Backdoor Attacks
• Can Large Language Models Find And Fix Vulnerable Software?
• How Can ChatGPT Support Human Security Testers to Help Mitigate Supply Chain Attacks?
• Identifying and Mitigating Privacy Risks Stemming from Language Models: A Survey
• Med-MMHL: A Multi-Modal Dataset for Detecting Human- and LLM-Generated Misinformation in the Medical Domain
• Practical Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibration
• An Efficient Membership Inference Attack for the Diffusion Model by Proximal Initialization
• Backdoor Attacks for In-Context Learning with Language Models
• Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Models
• Effective Prompt Extraction from Language Models
• Universal Jailbreak Backdoors from Poisoned Human Feedback
• REMARK-LLM: A Robust and Efficient Watermarking Framework for Generative Large Language Models
• Using Large Language Models for Cybersecurity Capture-The-Flag Challenges and Certification Questions
• DefectHunter: A Novel LLM-Driven Boosted-Conformer-based Code Vulnerability Detection Mechanism
• Probing Explicit and Implicit Gender Bias through LLM Conditional Text Generation
• Prompt Packer: Deceiving LLMs through Compositional Instruction with Hidden Attacks
• Source Attribution for Large Language Model-Generated Data
• Bot or Human? Detecting ChatGPT Imposters with A Single Question
• Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluations
• Privacy Side Channels in Machine Learning Systems
• On the Safety of Open-Sourced Large Language Models: Does Alignment Really Prevent Them From Being Misused?
• Using ChatGPT as a Static Application Security Testing Tool
• Self-Deception: Reverse Penetrating the Semantic Firewall of Large Language Models
• LoFT: Local Proxy Fine-tuning For Improving Transferability Of Adversarial Attacks Against Large Language Model
• A Theoretical Insight into Attack and Defense of Gradient Leakage in Transformer
• Test-time Backdoor Mitigation for Black-Box Large Language Models with Defensive Demonstrations
• VulLibGen: Generating Names of Vulnerability-Affected Packages via a Large Language Model
• A Probabilistic Fluctuation based Membership Inference Attack for Diffusion Models

Cited by in corpus (53)

• Unleashing the potential of prompt engineering for large language models
• AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathways
• Transformers and Large Language Models for Efficient Intrusion Detection Systems: A Comprehensive Survey
• Exploring the Capabilities and Limitations of Large Language Models in the Electric Energy Sector
• Tool Learning with Large Language Models: A Survey
• Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challenges
• Generative AI for Self-Adaptive Systems: State of the Art and Research Roadmap
• Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmap
• Exploring the Potential of Large Language Models for Improving Digital Forensic Investigation Efficiency
• Exploring Gen-AI applications in building research and industry: A review
• An Empirical Study of Challenges in Machine Learning Asset Management
• An overview of domain-specific foundation model: key technologies, applications and challenges
• Mapping the Landscape of Generative AI in Network Monitoring and Management
• Embodied AI with Foundation Models for Mobile Service Robots: A Systematic Review
• LLMs as Hackers: Autonomous Linux Privilege Escalation Attacks
• SoK: The Privacy Paradox of Large Language Models: Advancements, Privacy Risks, and Mitigation
• Understanding Users' Security and Privacy Concerns and Attitudes Towards Conversational AI Platforms
• VulScribeR: Exploring RAG-based Vulnerability Augmentation with LLMs
• Learning from Data Streams: An Overview and Update
• From Prediction to Explanation: Multimodal, Explainable, and Interactive Deepfake Detection Framework for Non-Expert Users
• Stylometry recognizes human and LLM-generated texts in short samples
• Advances in Artificial Intelligence: A Review for the Creative Industries
• Robustness and Cybersecurity in the EU Artificial Intelligence Act
• LLM-Driven APT Detection for 6G Wireless Networks: A Systematic Review and Taxonomy
• Crystalline Material Discovery in the Era of Artificial Intelligence
• Assessing the Effectiveness of LLMs in Android Application Vulnerability Analysis
• Religious Bias Landscape in Language and Text-to-Image Models: Analysis, Detection, and Debiasing Strategies
• LLLMs: A Data-Driven Survey of Evolving Research on Limitations of Large Language Models
• PILLAR: an AI-Powered Privacy Threat Modeling Tool
• SecMLOps: A Comprehensive Framework for Integrating Security Throughout the MLOps Lifecycle
• Comparing the Performance of LLMs in RAG-based Question-Answering: A Case Study in Computer Science Literature
• Verifying the Robustness of Automatic Credibility Assessment
• Large Reasoning Models Are Autonomous Jailbreak Agents
• Integrating Artificial Open Generative Artificial Intelligence into Software Supply Chain Security
• Using Large Language Models for Solving Thermodynamic Problems
• LLMs Have Rhythm: Fingerprinting Large Language Models Using Inter-Token Times and Network Traffic Analysis
• VFLAIR-LLM: A Comprehensive Framework and Benchmark for Split Learning of LLMs
• Large Language Model Deployment on Resource-Constrained Edge Devices: A Practitioners Survey
• Actionable Cybersecurity Notifications for Smart Homes: A User Study on the Role of Length and Complexity
• LM-Fix: Lightweight Bit-Flip Detection and Rapid Recovery Framework for Language Models
• SAFE: Advancing Large Language Models in Leveraging Semantic and Syntactic Relationships for Software Vulnerability Detection
• Exploring Potential Prompt Injection Attacks in Federated Military LLMs and Their Mitigation
• Federated Learning With Individualized Privacy Through Client Sampling
• REBot: From RAG to CatRAG with Semantic Enrichment and Graph Routing
• Context Conquers Parameters: Outperforming Proprietary LLM in Commit Message Generation
• Exploiting Latent Space Discontinuities for Building Universal LLM Jailbreaks and Data Extraction Attacks
• Transforming Remanufacturing Automation with Large Language Models: A Forward-Looking Analysis with Case Studies
• LLM-Augmented Knowledge Base Construction For Root Cause Analysis
• RAG-Enhanced LLMs for Interactive Explainability in Clinical Decision Support Systems
• PRISM-XR: Empowering Privacy-Aware XR Collaboration with Multimodal Large Language Models
• Trustworthy AI LLM Scalability Risk Index (LSRI): A Cybersecurity Framework Assessing Agentic-AI Security & Software Model Supply Chain Safety Boosting AI-Generated Malware Defense & Explainability Mitigating Emerging Risks of Generative AI
• Never say never: Exploring the effects of available knowledge on agent persuasiveness in controlled physiotherapy motivation dialogues
• Mitigating Sensitive Information Leakage in LLMs4Code through Machine Unlearning