Large Language Models Are Human-Level Prompt Engineers

Abstract

By conditioning on natural language instructions, large language models (LLMs) have displayed impressive capabilities as general-purpose computers. However, task performance depends significantly on the quality of the prompt used to steer the model, and most effective prompts have been handcrafted by humans. Inspired by classical program synthesis and the human approach to prompt engineering, we propose Automatic Prompt Engineer (APE) for automatic instruction generation and selection. In our method, we treat the instruction as the "program," optimized by searching over a pool of instruction candidates proposed by an LLM in order to maximize a chosen score function. To evaluate the quality of the selected instruction, we evaluate the zero-shot performance of another LLM following the selected instruction. Experiments on 24 NLP tasks show that our automatically generated instructions outperform the prior LLM baseline by a large margin and achieve better or comparable performance to the instructions generated by human annotators on 19/24 tasks. We conduct extensive qualitative and quantitative analyses to explore the performance of APE. We show that APE-engineered prompts can be applied to steer models toward truthfulness and/or informativeness, as well as to improve few-shot learning performance by simply prepending them to standard in-context learning prompts. Please check out our webpage at https://sites.google.com/view/automatic-prompt-engineer.

Cited by in corpus (21)

• Graph of Thoughts: Solving Elaborate Problems with Large Language Models
• Perspectives on Large Language Models for Relevance Judgment
• A GPT-4 Reticular Chemist for Guiding MOF Discovery
• Evaluating the Effectiveness of LLMs in Introductory Computer Science Education: A Semester-Long Field Study
• MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots
• Process Modeling With Large Language Models
• Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Guided Exploration for Zero-Shot Object Navigation
• A Practical Survey on Zero-shot Prompt Design for In-context Learning
• AppPoet: Large Language Model based Android malware detection via multi-view prompt engineering
• OntoChatGPT Information System: Ontology-Driven Structured Prompts for ChatGPT Meta-Learning
• Evaluating Large Language Models in Process Mining: Capabilities, Benchmarks, and Evaluation Strategies
• Large Knowledge Model: Perspectives and Challenges
• ProMoAI: Process Modeling with Generative AI
• Large Language Models, scientific knowledge and factuality: A framework to streamline human expert evaluation
• Evaluating Transfer Learning for Simplifying GitHub READMEs
• Scalable Multitask Learning Using Gradient-based Estimation of Task Affinity
• DataAgent: Evaluating Large Language Models' Ability to Answer Zero-Shot, Natural Language Queries
• Are NLP Models Good at Tracing Thoughts: An Overview of Narrative Understanding
• The Effects of Demographic Instructions on LLM Personas
• WebChecker: A Versatile EVL Plugin for Validating HTML Pages with Bootstrap Frameworks
• Simple and Effective Input Reformulations for Translation