Empirically-Calibrated H100 Node Power Models for Reducing Uncertainty in AI Training Energy Estimation

Abstract

As AI's energy demand continues to grow, it is critical to enhance the understanding of characteristics of this demand, to improve grid infrastructure planning and environmental assessment. By combining empirical measurements from Brookhaven National Laboratory during AI training on 8-GPU H100 systems with open-source benchmarking data, we develop statistical models relating computational intensity to node-level power consumption. We measure the gap between manufacturer-rated thermal design power (TDP) and actual power demand during AI training. Our analysis reveals that even computationally intensive workloads operate at only 76% of the 10.2 kW TDP rating. Our architecture-specific model, calibrated to floating-point operations, predicts energy consumption with 11.4% mean absolute percentage error, significantly outperforming TDP-based approaches (27-37% error). We identified distinct power signatures between transformer and CNN architectures, with transformers showing characteristic fluctuations that may impact grid stability.

4 figures, 22 pages

References in corpus (9)

• Compute and Energy Consumption Trends in Deep Learning Inference
• Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model
• Carbon Explorer: A Holistic Approach for Designing Carbon Aware Datacenters
• How to estimate carbon footprint when training deep learning models? A guide and review
• Trends in Energy Estimates for Computing in AI/Machine Learning Accelerators, Supercomputers, and Compute-Intensive Applications
• Verified Instruction-Level Energy Consumption Measurement for NVIDIA GPUs
• Optimizing performance per watt on GPUs in High Performance Computing: temperature, frequency and voltage effects
• Empirical Measurements of AI Training Power Demand on a GPU-Accelerated Node
• Energy and Carbon Considerations of Fine-Tuning BERT