Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study

Abstract

Large Language Models (LLMs), like ChatGPT, have demonstrated vast potential but also introduce challenges related to content constraints and potential misuse. Our study investigates three key research questions: (1) the number of different prompt types that can jailbreak LLMs, (2) the effectiveness of jailbreak prompts in circumventing LLM constraints, and (3) the resilience of ChatGPT against these jailbreak prompts. Initially, we develop a classification model to analyze the distribution of existing prompts, identifying ten distinct patterns and three categories of jailbreak prompts. Subsequently, we assess the jailbreak capability of prompts with ChatGPT versions 3.5 and 4.0, utilizing a dataset of 3,120 jailbreak questions across eight prohibited scenarios. Finally, we evaluate the resistance of ChatGPT against jailbreak prompts, finding that the prompts can consistently evade the restrictions in 40 use-case scenarios. The study underscores the importance of prompt structures in jailbreaking LLMs and discusses the challenges of robust jailbreak prompt generation and prevention.

Cited by in corpus (8)

• Bad Actor, Good Advisor: Exploring the Role of Large Language Models in Fake News Detection
• MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots
• FuzzLLM: A Novel and Universal Fuzzing Framework for Proactively Discovering Jailbreak Vulnerabilities in Large Language Models
• Silent Guardian: Protecting Text from Malicious Exploitation by Large Language Models
• Normative Conflicts and Shallow AI Alignment
• Prompt engineering for bibliographic web-scraping
• Humanoid Artificial Consciousness Designed with Large Language Model Based on Psychoanalysis and Personality Theory
• StyleRec: A Benchmark Dataset for Prompt Recovery in Writing Style Transformation