A Neural Network Solves, Explains, and Generates University Math Problems by Program Synthesis and Few-Shot Learning at Human Level

Abstract

We demonstrate that a neural network pre-trained on text and fine-tuned on code solves mathematics course problems, explains solutions, and generates new questions at a human level. We automatically synthesize programs using few-shot learning and OpenAI's Codex transformer and execute them to solve course problems at 81% automatic accuracy. We curate a new dataset of questions from MIT's largest mathematics courses (Single Variable and Multivariable Calculus, Differential Equations, Introduction to Probability and Statistics, Linear Algebra, and Mathematics for Computer Science) and Columbia University's Computational Linear Algebra. We solve questions from a MATH dataset (on Prealgebra, Algebra, Counting and Probability, Intermediate Algebra, Number Theory, and Precalculus), the latest benchmark of advanced mathematics problems designed to assess mathematical reasoning. We randomly sample questions and generate solutions with multiple modalities, including numbers, equations, and plots. The latest GPT-3 language model pre-trained on text automatically solves only 18.8% of these university questions using zero-shot learning and 30.8% using few-shot learning and the most recent chain of thought prompting. In contrast, program synthesis with few-shot learning using Codex fine-tuned on code generates programs that automatically solve 81% of these questions. Our approach improves the previous state-of-the-art automatic solution accuracy on the benchmark topics from 8.8% to 81.1%. We perform a survey to evaluate the quality and difficulty of generated questions. This work is the first to automatically solve university-level mathematics course questions at a human level and the first work to explain and generate university-level mathematics course questions at scale, a milestone for higher education.

181 pages, 8 figures, 280 tables

References in corpus (10)

• UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
• Language Models are Few-Shot Learners
• Evaluating Large Language Models Trained on Code
• Large Language Models are Zero-Shot Reasoners
• Measuring Massive Multitask Language Understanding
• Measuring Mathematical Problem Solving With the MATH Dataset
• Scaling Language Models: Methods, Analysis & Insights from Training Gopher
• A Neural Network Solves, Explains, and Generates University Math Problems by Program Synthesis and Few-Shot Learning at Human Level
• Solving Machine Learning Problems
• MWP-BERT: Numeracy-Augmented Pre-training for Math Word Problem Solving

Cited by in corpus (13)

• A Survey on Large Language Model based Autonomous Agents
• Practical and Ethical Challenges of Large Language Models in Education: A Systematic Scoping Review
• Using cognitive psychology to understand GPT-3
• Automatic Generation of Programming Exercises and Code Explanations using Large Language Models
• "It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers
• A Neural Network Solves, Explains, and Generates University Math Problems by Program Synthesis and Few-Shot Learning at Human Level
• The Rise of Artificial Intelligence in Educational Measurement: Opportunities and Ethical Challenges
• LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Provers
• Can neural networks do arithmetic? A survey on the elementary numerical skills of state-of-the-art deep learning models
• Symbolic Equation Solving via Reinforcement Learning
• Complex QA and language models hybrid architectures, Survey
• Decomposed Prompting to Answer Questions on a Course Discussion Board
• The Liouville Generator for Producing Integrable Expressions