HetSeq: Distributed GPU Training on Heterogeneous Infrastructure

Yifan Ding, Nicholas Botzer, Tim Weninger

Modern deep learning systems like PyTorch and Tensorflow are able to train enormous models with billions (or trillions) of parameters on a distributed infrastructure. These systems…