Show Your Work: Scratchpads for Intermediate Computation with Language Models

Abstract

Large pre-trained language models perform remarkably well on tasks that can be done "in one pass", such as generating realistic text or synthesizing computer programs. However, they struggle with tasks that require unbounded multi-step computation, such as adding integers or executing programs. Surprisingly, we find that these same models are able to perform complex multi-step computations -- even in the few-shot regime -- when asked to perform the operation "step by step", showing the results of intermediate computations. In particular, we train transformers to perform multi-step computations by asking them to emit intermediate computation steps into a "scratchpad". On a series of increasingly complex tasks ranging from long addition to the execution of arbitrary programs, we show that scratchpads dramatically improve the ability of language models to perform multi-step computations.

References in corpus (11)

• Language Models are Few-Shot Learners
• Learning to Map Sentences to Logical Form: Structured Classification with Probabilistic Categorial Grammars
• Universal Transformers
• Adaptive Computation Time for Recurrent Neural Networks
• Neural Programmer-Interpreters
• Neural GPUs Learn Algorithms
• RobustFill: Neural Program Learning under Noisy I/O
• Pretrained Transformers as Universal Computation Engines
• Program Synthesis with Large Language Models
• PonderNet: Learning to Ponder
• Neural Random-Access Machines

Cited by in corpus (1)

• Can neural networks do arithmetic? A survey on the elementary numerical skills of state-of-the-art deep learning models