Automatic Graph Partitioning for Very Large-scale Deep Learning

Masahiro Tanaka, Kenjiro Taura, Toshihiro Hanawa +1

This work proposes RaNNC (Rapid Neural Network Connector) as middleware for automatic hybrid parallelism. In recent deep learning research, as exemplified by T5 and GPT-3, the size…