The Cost of Training NLP Models: A Concise Overview

Abstract

We review the cost of training large-scale language models, and the drivers of these costs. The intended audience includes engineers and scientists budgeting their model-training experiments, as well as non-practitioners trying to make sense of the economics of modern-day Natural Language Processing (NLP).

References in corpus (2)

• Scaling Laws for Neural Language Models
• TinyBERT: Distilling BERT for Natural Language Understanding

Cited by in corpus (20)

• Challenges in Deploying Machine Learning: a Survey of Case Studies
• Compute Trends Across Three Eras of Machine Learning
• Is Neuro-Symbolic AI Meeting its Promise in Natural Language Processing? A Structured Review
• Mokey: Enabling Narrow Fixed-Point Inference for Out-of-the-Box Floating-Point Transformer Models
• First-Generation Inference Accelerator Deployment at Facebook
• When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset
• Accenture at CheckThat! 2020: If you say so: Post-hoc fact-checking of claims using transformer-based models
• GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Training
• Scaling Down Deep Learning with MNIST-1D
• Automated Essay Scoring Using Transformer Models
• Complex QA and language models hybrid architectures, Survey
• Surprisal-Triggered Conditional Computation with Neural Networks
• Artificial neurons based on antiferromagnetic auto-oscillators as a platform for neuromorphic computing
• Varuna: Scalable, Low-cost Training of Massive Deep Learning Models
• Using Artificial Populations to Study Psychological Phenomena in Neural Models
• Exceeding the Limits of Visual-Linguistic Multi-Task Learning
• Technical Report: Auxiliary Tuning and its Application to Conditional Text Generation
• Consistent Accelerated Inference via Confident Adaptive Transformers
• The Efficiency Misnomer
• On the Existence of Universal Lottery Tickets