What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Use
arXiv:2409.08775 · doi:10.1145/3731756
Abstract
Prompting LLMs for complex tasks (e.g., building a trip advisor chatbot) needs humans to clearly articulate customized requirements (e.g., "start the response with a tl;dr"). However, existing prompt engineering instructions often lack focused training on requirement articulation and instead tend to emphasize increasingly automatable strategies (e.g., tricks like adding role-plays and "think step-by-step"). To address the gap, we introduce Requirement-Oriented Prompt Engineering (ROPE), a paradigm that focuses human attention on generating clear, complete requirements during prompting. We implement ROPE through an assessment and training suite that provides deliberate practice with LLM-generated feedback. In a randomized controlled experiment with 30 novices, ROPE significantly outperforms conventional prompt engineering training (20% vs. 1% gains), a gap that automatic prompt optimization cannot close. Furthermore, we demonstrate a direct correlation between the quality of input requirements and LLM outputs. Our work paves the way to empower more end-users to build complex LLM applications.
15 pages; TOCHI 2025
References in corpus (14)
- Training language models to follow instructions with human feedback
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- "It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers
- An Empirical Study of the Non-determinism of ChatGPT in Code Generation
- ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing
- "What It Wants Me To Say": Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Models
- PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinement
- EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteria
- How to Teach Programming in the AI Era? Using LLMs as a Teachable Agent for Debugging
- How Beginning Programmers and Code LLMs (Mis)read Each Other
- Learning Agent-based Modeling with LLM Companions: Experiences of Novices and Experts Using ChatGPT & NetLogo Chat
- SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
- CoPrompt: Supporting Prompt Sharing and Referring in Collaborative Natural Language Programming
- Interactive Code Generation via Test-Driven User-Intent Formalization