Efficient Fine-Grained GPU Performance Modeling for Distributed Deep Learning of LLM

Biyao Zhang, Mingkai Zheng, Debargha Ganguly +4

Training Large Language Models(LLMs) is one of the most compute-intensive tasks in high-performance computing. Predicting end-to-end training time for multi-billion parameter model…