Survey on Large Scale Neural Network Training

Julia Gusak, Daria Cherniuk, Alena Shilova +8

Modern Deep Neural Networks (DNNs) require significant memory to store weight, activations, and other intermediate tensors during training. Hence, many models do not fit one GPU de…