SuperScaler: Supporting Flexible DNN Parallelization via a Unified Abstraction

Zhiqi Lin, Youshan Miao, Guodong Liu +10

With the growing model size, deep neural networks (DNN) are increasingly trained over massive GPU accelerators, which demands a proper parallelization plan that transforms a DNN mo…