The Cost of Training NLP Models: A Concise Overview

Abstract

We review the cost of training large-scale language models, and the drivers of these costs. The intended audience includes engineers and scientists budgeting their model-training experiments, as well as non-practitioners trying to make sense of the economics of modern-day Natural Language Processing (NLP).

References in corpus (1)

• Scaling Laws for Neural Language Models

Cited by in corpus (6)

• First-Generation Inference Accelerator Deployment at Facebook
• Accenture at CheckThat! 2020: If you say so: Post-hoc fact-checking of claims using transformer-based models
• Surprisal-Triggered Conditional Computation with Neural Networks
• Artificial neurons based on antiferromagnetic auto-oscillators as a platform for neuromorphic computing
• Technical Report: Auxiliary Tuning and its Application to Conditional Text Generation
• Exceeding the Limits of Visual-Linguistic Multi-Task Learning