DeepCompile: A Compiler-Driven Approach to Optimizing Distributed Deep Learning Training

Masahiro Tanaka, Du Li, Umesh Chand +3

The rapid growth of deep learning models has increased the demand for efficient distributed training strategies. Fully sharded approaches like ZeRO-3 and FSDP partition model param…