Least-to-Most Prompting Enables Complex Reasoning in Large Language Models

Abstract

Chain-of-thought prompting has demonstrated remarkable performance on various natural language reasoning tasks. However, it tends to perform poorly on tasks which requires solving problems harder than the exemplars shown in the prompts. To overcome this challenge of easy-to-hard generalization, we propose a novel prompting strategy, least-to-most prompting. The key idea in this strategy is to break down a complex problem into a series of simpler subproblems and then solve them in sequence. Solving each subproblem is facilitated by the answers to previously solved subproblems. Our experimental results on tasks related to symbolic manipulation, compositional generalization, and math reasoning reveal that least-to-most prompting is capable of generalizing to more difficult problems than those seen in the prompts. A notable finding is that when the GPT-3 code-davinci-002 model is used with least-to-most prompting, it can solve the compositional generalization benchmark SCAN in any split (including length split) with an accuracy of at least 99% using just 14 exemplars, compared to only 16% accuracy with chain-of-thought prompting. This is particularly noteworthy because neural-symbolic models in the literature that specialize in solving SCAN are trained on the entire training set containing over 15,000 examples. We have included prompts for all the tasks in the Appendix.

ICLR 2023

Cited by in corpus (16)

• Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses
• ChartGPT: Leveraging LLMs to Generate Charts from Abstract Natural Language
• LLM for SoC Security: A Paradigm Shift
• Enhancing Knowledge Retrieval with In-Context Learning and Semantic Search through Generative AI
• ThoughtSource: A central hub for large language model reasoning data
• Safety Analysis in the Era of Large Language Models: A Case Study of STPA using ChatGPT
• eaSEL: Promoting Social-Emotional Learning and Parent-Child Interaction through AI-Mediated Content Consumption
• ProMoAI: Process Modeling with Generative AI
• The Life Cycle of Knowledge in Big Language Models: A Survey
• Criteria-Based LLM Relevance Judgments
• Complex QA and language models hybrid architectures, Survey
• Concept-Guided Chain-of-Thought Prompting for Pairwise Comparison Scoring of Texts with Large Language Models
• The meaning of prompts and the prompts of meaning: Semiotic reflections and modelling
• Machine Reading Comprehension using Case-based Reasoning
• Rethinking the Chain-of-Thought: The Roles of In-Context Learning and Pre-trained Priors
• Simple and Effective Input Reformulations for Translation