Training Verifiers to Solve Math Word Problems
arXiv:2110.14168
Abstract
State-of-the-art language models can match human performance on many tasks, but they still struggle to robustly perform multi-step mathematical reasoning. To diagnose the failures of current models and support research, we introduce GSM8K, a dataset of 8.5K high quality linguistically diverse grade school math word problems. We find that even the largest transformer models fail to achieve high test performance, despite the conceptual simplicity of this problem distribution. To increase performance, we propose training verifiers to judge the correctness of model completions. At test time, we generate many candidate solutions and select the one ranked highest by the verifier. We demonstrate that verification significantly improves performance on GSM8K, and we provide strong empirical evidence that verification scales more effectively with increased data than a finetuning baseline.
References in corpus (15)
- Sequence to Sequence Learning with Neural Networks
- Language Models are Few-Shot Learners
- Scaling Laws for Neural Language Models
- CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge
- Measuring Mathematical Problem Solving With the MATH Dataset
- MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms
- MathBERT: A Pre-Trained Model for Mathematical Formula Understanding
- Deep Learning for Symbolic Mathematics
- Ape210K: A Large-Scale and Template-Rich Dataset of Math Word Problems
- Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problems
- Mapping Natural-language Problems to Formal-language Solutions Using Structured Neural Representations
- MathBERT: A Pre-trained Language Model for General NLP Tasks in Mathematics Education
- Semantically-Aligned Equation Generation for Solving and Reasoning Math Word Problems
- A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers
- Generate & Rank: A Multi-task Framework for Math Word Problems