Measuring Massive Multitask Language Understanding

Abstract

We propose a new test to measure a text model's multitask accuracy. The test covers 57 tasks including elementary mathematics, US history, computer science, law, and more. To attain high accuracy on this test, models must possess extensive world knowledge and problem solving ability. We find that while most recent models have near random-chance accuracy, the very largest GPT-3 model improves over random chance by almost 20 percentage points on average. However, on every one of the 57 tasks, the best models still need substantial improvements before they can reach expert-level accuracy. Models also have lopsided performance and frequently do not know when they are wrong. Worse, they still have near-random accuracy on some socially important subjects such as morality and law. By comprehensively evaluating the breadth and depth of a model's academic and professional understanding, our test can be used to analyze models across many tasks and to identify important shortcomings.

arXiv commentICLR 2021; the test and code is available at https://github.com/hendrycks/test

References in corpus (7)

• Language Models are Few-Shot Learners
• Shortcut Learning in Deep Neural Networks
• On Calibration of Modern Neural Networks
• Scaling Laws for Neural Language Models
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• Deep Anomaly Detection with Outlier Exposure
• HellaSwag: Can a Machine Really Finish Your Sentence?

Cited by in corpus (26)

• A Neural Network Solves, Explains, and Generates University Math Problems by Program Synthesis and Few-Shot Learning at Human Level
• What Large Language Models Know and What People Think They Know
• Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses
• Large language models surpass human experts in predicting neuroscience results
• Can GPT-4 Support Analysis of Textual Data in Tasks Requiring Highly Specialized Domain Expertise?
• Can Generative Pre-trained Transformers (GPT) Pass Assessments in Higher Education Programming Courses?
• OpenMedLM: Prompt engineering can out-perform fine-tuning in medical question-answering with open-source large language models
• A Scoping Review of ChatGPT Research in Accounting and Finance
• Designing Heterogeneous LLM Agents for Financial Sentiment Analysis
• ThoughtSource: A central hub for large language model reasoning data
• OntoChatGPT Information System: Ontology-Driven Structured Prompts for ChatGPT Meta-Learning
• The Future of Intelligent Healthcare: A Systematic Analysis and Discussion on the Integration and Impact of Robots Using Large Language Models for Healthcare
• Clinical Insights: A Comprehensive Review of Language Models in Medicine
• Domain-specific ChatBots for Science using Embeddings
• Applications of Generative AI in Healthcare: algorithmic, ethical, legal and societal considerations
• An empirical study of LLaMA3 quantization: from LLMs to MLLMs
• CIPHER: Cybersecurity Intelligent Penetration-testing Helper for Ethical Researcher
• Benchmarking large language models for materials synthesis: the case of atomic layer deposition
• Transforming Agency. On the mode of existence of Large Language Models
• Truthful AI: Developing and governing AI that does not lie
• Helpful assistant or fruitful facilitator? Investigating how personas affect language model behavior
• NoRe: Augmenting Journaling Experience with Generative AI for Music Creation
• Identifying Query-Relevant Neurons in Large Language Models for Long-Form Texts
• Neuralink: Fast LLM Inference on Smartphones with Neuron Co-Activation Linking
• Enhancing AI Safety Through the Fusion of Low Rank Adapters
• SCOPE: Real-Time Natural Language Camera Agent at the Edge