Scaling Language Models: Methods, Analysis & Insights from Training Gopher

Abstract

Language modelling provides a step towards intelligent communication systems by harnessing large repositories of written human knowledge to better predict and understand the world. In this paper, we present an analysis of Transformer-based language model performance across a wide range of model scales -- from models with tens of millions of parameters up to a 280 billion parameter model called Gopher. These models are evaluated on 152 diverse tasks, achieving state-of-the-art performance across the majority. Gains from scale are largest in areas such as reading comprehension, fact-checking, and the identification of toxic language, but logical and mathematical reasoning see less benefit. We provide a holistic analysis of the training dataset and model's behaviour, covering the intersection of model scale with bias and toxicity. Finally we discuss the application of language models to AI safety and the mitigation of downstream harms.

arXiv comment120 pages

Cited by in corpus (19)

• Co-Writing with Opinionated Language Models Affects Users' Views
• A Neural Network Solves, Explains, and Generates University Math Problems by Program Synthesis and Few-Shot Learning at Human Level
• Uncovering ChatGPT's Capabilities in Recommender Systems
• Harms from Increasingly Agentic Algorithmic Systems
• KICGPT: Large Language Model with Knowledge in Context for Knowledge Graph Completion
• Astronomia ex machina: a history, primer, and outlook on neural networks in astronomy
• State-of-the-art generalisation research in NLP: A taxonomy and review
• Linguistically inspired roadmap for building biologically reliable protein language models
• Defining the scope of AI regulations
• A Survey on Symbolic Knowledge Distillation of Large Language Models
• Comparing Sentence-Level Suggestions to Message-Level Suggestions in AI-Mediated Communication
• A Holistic Assessment of the Carbon Footprint of Noor, a Very Large Arabic Language Model
• TextDescriptives: A Python package for calculating a large variety of metrics from text
• A Comprehensive Review of State-of-The-Art Methods for Java Code Generation from Natural Language Text
• Findings of the The RuATD Shared Task 2022 on Artificial Text Detection in Russian
• Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Model
• CoRRPUS: Code-based Structured Prompting for Neurosymbolic Story Understanding
• Emergent inabilities? Inverse scaling over the course of pretraining
• CURE: A Dataset for Clinical Understanding & Retrieval Evaluation