Pairing Analogy-Augmented Generation with Procedural Memory for Procedural Q&A

Abstract

Large language models struggle to synthesize disparate pieces of information into a coherent plan when approaching a complex procedural task. In this work, we introduce a novel formalism and structure for such procedural knowledge. Based on this formalism, we present a novel procedural knowledge dataset called LCStep, which we created from LangChain tutorials. To leverage this procedural knowledge to solve new tasks, we propose analogy-augmented generation (AAG), which draws inspiration from the human ability to assimilate past experiences to solve unfamiliar problems. AAG uses a custom procedure memory store to retrieve and adapt specialized domain knowledge to answer new procedural tasks. We demonstrate that AAG outperforms few-shot and RAG baselines on LCStep, RecipeNLG, and CHAMP datasets under a pairwise LLM-based evaluation, corroborated by human evaluation in the case of RecipeNLG.

References in corpus (7)

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• A Survey on Large Language Model based Autonomous Agents
• Retrieval-Augmented Generation for Large Language Models: A Survey
• ReAct: Synergizing Reasoning and Acting in Language Models
• Self-critiquing models for assisting human evaluators
• Large Language Models as Analogical Reasoners
• PlaSma: Making Small Language Models Better Procedural Knowledge Models for (Counterfactual) Planning