Reducing Energy Bloat in Large Model Training

Abstract

Training large AI models on numerous GPUs consumes a massive amount of energy, making power delivery one of the largest limiting factors in building and operating datacenters for AI workloads. However, we observe that not all energy consumed during training directly contributes to end-to-end throughput; a significant portion can be removed without slowing down training. We call this portion energy bloat. In this work, we identify two independent sources of energy bloat in large model training and propose Perseus, a training system that mitigates both. To do this, Perseus obtains the time--energy tradeoff frontier of a large model training job using an efficient graph cut-based algorithm, and schedules computation energy consumption across time to reduce both types of energy bloat. Evaluation on large models, including GPT-3 and Bloom, shows that Perseus reduces the energy consumption of large model training by up to 30% without any throughput loss or hardware modification.

arXiv commentSOSP 24 | Open-source part of Zeus at https://ml.energy/zeus/research_overview/perseus/

References in corpus (20)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Training Deep Nets with Sublinear Memory Cost
• Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
• GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
• Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model
• Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning
• Serving DNNs like Clockwork: Performance Predictability from the Bottom Up
• Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads
• Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Models
• Memory-Efficient Pipeline-Parallel DNN Training
• Dynamic GPU Energy Optimization for Machine Learning Training Workloads
• Oobleck: Resilient Distributed Training of Large Models Using Pipeline Templates
• DAPPLE: A Pipelined Data Parallel Approach for Training Large Models
• MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
• Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Training
• Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs
• PipeFisher: Efficient Training of Large Language Models Using Pipelining and Fisher Information Matrices
• Chasing Low-Carbon Electricity for Practical and Sustainable DNN Training

Cited by in corpus (1)

• ReGate: Enabling Power Gating in Neural Processing Units