Ignore Previous Prompt: Attack Techniques For Language Models
arXiv:2211.09527
Abstract
Transformer-based large language models (LLMs) provide a powerful foundation for natural language tasks in large-scale customer-facing applications. However, studies that explore their vulnerabilities emerging from malicious user interaction are scarce. By proposing PromptInject, a prosaic alignment framework for mask-based iterative adversarial prompt composition, we examine how GPT-3, the most widely deployed language model in production, can be easily misaligned by simple handcrafted inputs. In particular, we investigate two types of attacks -- goal hijacking and prompt leaking -- and demonstrate that even low-aptitude, but sufficiently ill-intentioned agents, can easily exploit GPT-3's stochastic nature, creating long-tail risks. The code for PromptInject is available at https://github.com/agencyenterprise/PromptInject.
arXiv commentML Safety Workshop NeurIPS 2022
Cited by in corpus (6)
- A Survey on ChatGPT: AI-Generated Contents, Challenges, and Solutions
- MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots
- Is GPT-4 a reliable rater? Evaluating Consistency in GPT-4 Text Ratings
- Embedding Democratic Values into Social Media AIs via Societal Objective Functions
- Building Privacy-Preserving and Secure Geospatial Artificial Intelligence Foundation Models
- Target-driven Attack for Large Language Models