The Joule Point: an Energy-Optimal Operating Point for AI Inference

Alexander Apartsin, Yehudit Aperstein

Data centers serving AI inference strive to maximize GPU utilization, running their cards at full power by default: this maximizes throughput and holds latencies down, but it is en…