Carbon Emissions and Large Neural Network Training

Abstract

The computation demand for machine learning (ML) has grown rapidly recently, which comes with a number of costs. Estimating the energy cost helps measure its environmental impact and finding greener strategies, yet it is challenging without detailed information. We calculate the energy use and carbon footprint of several recent large models-T5, Meena, GShard, Switch Transformer, and GPT-3-and refine earlier estimates for the neural architecture search that found Evolved Transformer. We highlight the following opportunities to improve energy efficiency and CO2 equivalent emissions (CO2e): Large but sparsely activated DNNs can consume <1/10th the energy of large, dense DNNs without sacrificing accuracy despite using as many or even more parameters. Geographic location matters for ML workload scheduling since the fraction of carbon-free energy and resulting CO2e vary ~5X-10X, even within the same country and the same organization. We are now optimizing where and when large models are trained. Specific datacenter infrastructure matters, as Cloud datacenters can be ~1.4-2X more energy efficient than typical datacenters, and the ML-oriented accelerators inside them can be ~2-5X more effective than off-the-shelf systems. Remarkably, the choice of DNN, datacenter, and processor can reduce the carbon footprint up to ~100-1000X. These large factors also make retroactive estimates of energy cost difficult. To avoid miscalculations, we believe ML papers requiring large computational resources should make energy consumption and CO2e explicit when practical. We are working to be more transparent about energy use and CO2e in our future research. To help reduce the carbon footprint of ML, we believe energy usage and CO2e should be a key metric in evaluating models, and we are collaborating with MLPerf developers to include energy usage during training and inference in this industry standard benchmark.

References in corpus (7)

• Distilling the Knowledge in a Neural Network
• DARTS: Differentiable Architecture Search
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
• The Evolved Transformer
• Quantifying the Carbon Emissions of Machine Learning
• Measuring the Algorithmic Efficiency of Neural Networks

Cited by in corpus (42)

• On the Opportunities and Risks of Foundation Models
• Evaluating Large Language Models Trained on Code
• Deep physical neural networks enabled by a backpropagation algorithm for arbitrary physical systems
• Multitask Prompted Training Enables Zero-Shot Task Generalization
• Sustainable AI: Environmental Implications, Challenges and Opportunities
• Factuality Challenges in the Era of Large Language Models
• Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
• Training Deep Boltzmann Networks with Sparse Ising Machines
• Towards Green Automated Machine Learning: Status Quo and Future Directions
• Towards Continual Knowledge Learning of Language Models
• Exploring the Limits of Large Scale Pre-training
• Scaling Vision with Sparse Mixture of Experts
• Mind the Gap: Assessing Temporal Generalization in Neural Language Models
• Why we need biased AI -- How including cognitive and ethical machine biases can enhance AI systems
• M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining
• The Stability-Efficiency Dilemma: Investigating Sequence Length Warmup for Training GPT Models
• How Powerful are Performance Predictors in Neural Architecture Search?
• The MultiBERTs: BERT Reproductions for Robustness Analysis
• Carbon Containers: A System-level Facility for Managing Application-level Carbon Emissions
• Sparse Training via Boosting Pruning Plasticity with Neuroregeneration
• BoolNet: Minimizing The Energy Consumption of Binary Neural Networks
• Primer: Searching for Efficient Transformers for Language Modeling
• Procedural Content Generation: Better Benchmarks for Transfer Reinforcement Learning
• Full-Cycle Energy Consumption Benchmark for Low-Carbon Computer Vision
• Perhaps PTLMs Should Go to School -- A Task to Assess Open Book and Closed Book QA
• Eco-Aware Graph Neural Networks for Sustainable Recommendations
• A Loss Curvature Perspective on Training Instability in Deep Learning
• Socio-Technological Challenges and Opportunities: Paths Forward
• Curb Your Carbon Emissions: Benchmarking Carbon Emissions in Machine Translation
• What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers
• Coupled Gradient Estimators for Discrete Latent Variables
• Greenformer: Factorization Toolkit for Efficient Deep Neural Networks
• Towards Structured Dynamic Sparse Pre-Training of BERT
• Pre-train or Annotate? Domain Adaptation with a Constrained Budget
• Prune Your Model Before Distill It
• SVP-CF: Selection via Proxy for Collaborative Filtering Data
• Toward Foundation Models for Earth Monitoring: Proposal for a Climate Change Benchmark
• An Overview on Generative AI at Scale with Edge-Cloud Computing
• Sparse MoEs meet Efficient Ensembles
• The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation
• The Efficiency Misnomer
• Structured Context and High-Coverage Grammar for Conversational Question Answering over Knowledge Graphs