Measuring Mathematical Problem Solving With the MATH Dataset

Abstract

Many intellectual endeavors require mathematical problem solving, but this skill remains beyond the capabilities of computers. To measure this ability in machine learning models, we introduce MATH, a new dataset of 12,500 challenging competition mathematics problems. Each problem in MATH has a full step-by-step solution which can be used to teach models to generate answer derivations and explanations. To facilitate future research and increase accuracy on MATH, we also contribute a large auxiliary pretraining dataset which helps teach models the fundamentals of mathematics. Even though we are able to increase accuracy on MATH, our results show that accuracy remains relatively low, even with enormous Transformer models. Moreover, we find that simply increasing budgets and model parameter counts will be impractical for achieving strong mathematical reasoning if scaling trends continue. While scaling Transformers is automatically solving most other text-based tasks, scaling is not currently solving MATH. To have more traction on mathematical problem solving we will likely need new algorithmic advancements from the broader research community.

arXiv commentNeurIPS 2021. Code and the MATH dataset is available at https://github.com/hendrycks/math/

References in corpus (15)

• Sequence to Sequence Learning with Neural Networks
• Decoupled Weight Decay Regularization
• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
• Scaling Laws for Neural Language Models
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention
• Scaling Laws for Autoregressive Generative Modeling
• Analysing Mathematical Reasoning Abilities of Neural Models
• GamePad: A Learning Environment for Theorem Proving
• Generative Language Modeling for Automated Theorem Proving
• Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problems
• Program Synthesis with Large Language Models
• Scaling Laws for Transfer
• Improving Graph Neural Network Representations of Logical Formulae with Subgraph Pooling
• LIME: Learning Inductive Bias for Primitives of Mathematical Reasoning
• MathZero, The Classification Problem, and Set-Theoretic Type Theory

Cited by in corpus (23)

• A Neural Network Solves, Explains, and Generates University Math Problems by Program Synthesis and Few-Shot Learning at Human Level
• Measuring Coding Challenge Competence With APPS
• How understanding large language models can inform the use of ChatGPT in physics education
• CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review
• Can neural networks do arithmetic? A survey on the elementary numerical skills of state-of-the-art deep learning models
• Large Knowledge Model: Perspectives and Challenges
• MiniF2F: a cross-system benchmark for formal Olympiad-level mathematics
• Training Verifiers to Solve Math Word Problems
• Effect of Pre-Training Scale on Intra- and Inter-Domain Full and Few-Shot Transfer Learning for Natural and Medical X-Ray Chest Images
• Spiers Memorial Lecture: How to do impactful research in artificial intelligence for chemistry and materials science
• NaturalProofs: Mathematical Theorem Proving in Natural Language
• Solving Machine Learning Problems
• OPT-R: Exploring the Role of Explanations in Finetuning and Prompting for Reasoning Skills of Large Language Models
• Solving Probability and Statistics Problems by Program Synthesis
• The ADAIO System at the BEA-2023 Shared Task on Generating AI Teacher Responses in Educational Dialogues
• AutoMathKG: The automated mathematical knowledge graph based on LLM and vector database
• Target-driven Attack for Large Language Models
• The Liouville Generator for Producing Integrable Expressions
• Towards Tractable Mathematical Reasoning: Challenges, Strategies, and Opportunities for Solving Math Word Problems
• Teaching Autoregressive Language Models Complex Tasks By Demonstration
• The Computational Complexity of Finding Arithmetic Expressions With and Without Parentheses
• Dynamic Framework for Collaborative Learning: Leveraging Advanced LLM with Adaptive Feedback Mechanisms
• Pretrained Language Models are Symbolic Mathematics Solvers too!