Teaching Autoregressive Language Models Complex Tasks By Demonstration
arXiv:2109.02102
Abstract
This paper demonstrates that by fine-tuning an autoregressive language model (GPT-Neo) on appropriately structured step-by-step demonstrations, it is possible to teach it to execute a mathematical task that has previously proved difficult for Transformers - longhand modulo operations - with a relatively small number of examples. Specifically, we fine-tune GPT-Neo to solve the numbers__div_remainder task from the DeepMind Mathematics Dataset; Saxton et al. (arXiv:1904.01557) reported below 40% accuracy on this task with 2 million training examples. We show that after fine-tuning on 200 appropriately structured demonstrations of solving long division problems and reporting the remainders, the smallest available GPT-Neo model achieves over 80% accuracy. This is achieved by constructing an appropriate dataset for fine-tuning, with no changes to the learning algorithm. These results suggest that fine-tuning autoregressive language models on small sets of well-crafted demonstrations may be a useful paradigm for enabling individuals without training in machine learning to coax such models to perform some kinds of complex multi-step tasks.
Corrected typo in Figure 2. Updated two citations to adhere to the format preferred by the cited authors
References in corpus (14)
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Snorkel: Rapid Training Data Creation with Weak Supervision
- Decision Transformer: Reinforcement Learning via Sequence Modeling
- Professor Forcing: A New Algorithm for Training Recurrent Networks
- Prefix-Tuning: Optimizing Continuous Prompts for Generation
- Measuring Mathematical Problem Solving With the MATH Dataset
- Scaling Laws for Autoregressive Generative Modeling
- Analysing Mathematical Reasoning Abilities of Neural Models
- Multimodal Few-Shot Learning with Frozen Language Models
- Neural Module Networks for Reasoning over Text
- Imitating Interactive Intelligence
- Investigating the Limitations of Transformers with Simple Arithmetic Tasks
- Attention Forcing for Sequence-to-sequence Model Training
- A Reinforcement Learning Environment for Mathematical Reasoning via Program Synthesis