Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model

Abstract

Progress in machine learning (ML) comes with a cost to the environment, given that training ML models requires significant computational resources, energy and materials. In the present article, we aim to quantify the carbon footprint of BLOOM, a 176-billion parameter language model, across its life cycle. We estimate that BLOOM's final training emitted approximately 24.7 tonnes of~\carboneq~if we consider only the dynamic power consumption, and 50.5 tonnes if we account for all processes ranging from equipment manufacturing to energy-based operational consumption. We also study the energy requirements and carbon emissions of its deployment for inference via an API endpoint receiving user queries in real-time. We conclude with a discussion regarding the difficulty of precisely estimating the carbon footprint of ML models and future research directions that can contribute towards improving carbon emissions reporting.

Cited by in corpus (10)

• Worldwide AI Ethics: a review of 200 guidelines and recommendations for AI governance
• Opening up ChatGPT: Tracking openness, transparency, and accountability in instruction-tuned text generators
• AI Art is Theft: Labour, Extraction, and Exploitation, Or, On the Dangers of Stochastic Pollocks
• Reducing Energy Bloat in Large Model Training
• Energy and Carbon Considerations of Fine-Tuning BERT
• Efficiency is Not Enough: A Critical Perspective of Environmentally Sustainable AI
• Tucano: Advancing Neural Text Generation for Portuguese
• Greening AI-enabled Systems with Software Engineering: A Research Agenda for Environmentally Sustainable AI Practices
• Empirically-Calibrated H100 Node Power Models for Reducing Uncertainty in AI Training Energy Estimation
• Breaking the ICE: Exploring promises and challenges of benchmarks for Inference Carbon & Energy estimation for LLMs