Data-parallel distributed training of very large models beyond GPU capacity

Samuel Matzek, Max Grossman, Minsik Cho +3

GPUs have limited memory and it is difficult to train wide and/or deep models that cause the training process to go out of memory. It is shown in this paper how an open source tool…