Beyond the Memory Wall: A Case for Memory-centric HPC System for Deep Learning

Abstract

As the models and the datasets to train deep learning (DL) models scale, system architects are faced with new challenges, one of which is the memory capacity bottleneck, where the limited physical memory inside the accelerator device constrains the algorithm that can be studied. We propose a memory-centric deep learning system that can transparently expand the memory capacity available to the accelerators while also providing fast inter-device communication for parallel training. Our proposal aggregates a pool of memory modules locally within the device-side interconnect, which are decoupled from the host interface and function as a vehicle for transparent memory capacity expansion. Compared to conventional systems, our proposal achieves an average 2.8x speedup on eight DL applications and increases the system-wide memory capacity to tens of TBs.

Published as a conference paper at the 51st IEEE/ACM International Symposium on Microarchitecture (MICRO-51), 2018

References in corpus (7)

• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• One weird trick for parallelizing convolutional neural networks
• Training Deep Nets with Sublinear Memory Cost
• Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes
• Dynamic Neural Turing Machine with Soft and Hard Addressing Schemes
• On Scale-out Deep Learning Training for Cloud and HPC