ResIST: Layer-Wise Decomposition of ResNets for Distributed Training

Chen Dun, Cameron R. Wolfe, Christopher M. Jermaine +1

We propose ResIST, a novel distributed training protocol for Residual Networks (ResNets). ResIST randomly decomposes a global ResNet into several shallow sub-ResNets that are train…