A Runtime-Based Computational Performance Predictor for Deep Neural Network Training
arXiv:2102.00527
Abstract
Deep learning researchers and practitioners usually leverage GPUs to help train their deep neural networks (DNNs) faster. However, choosing which GPU to use is challenging both because (i) there are many options, and (ii) users grapple with competing concerns: maximizing compute performance while minimizing costs. In this work, we present a new practical technique to help users make informed and cost-efficient GPU selections: make performance predictions with the help of a GPU that the user already has. Our technique exploits the observation that, because DNN training consists of repetitive compute steps, predicting the execution time of a single iteration is usually enough to characterize the performance of an entire training process. We make predictions by scaling the execution time of each operation in a training iteration from one GPU to another using either (i) wave scaling, a technique based on a GPU's execution model, or (ii) pre-trained multilayer perceptrons. We implement our technique into a Python library called Habitat and find that it makes accurate iteration execution time predictions (with an average error of 11.8%) on ResNet-50, Inception v3, the Transformer, GNMT, and DCGAN across six different GPU architectures. Habitat supports PyTorch, is easy to use, and is open source.
arXiv comment19 pages, 7 figures. Appears in the Proceedings of the 2021 USENIX Annual Technical Conference (USENIX ATC '21). Code available at https://github.com/geoffxy/habitat
References in corpus (12)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Sequence to Sequence Learning with Neural Networks
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
- cuDNN: Efficient Primitives for Deep Learning
- Mixed Precision Training
- Learning to Optimize Tensor Programs
- Fathom: Reference Workloads for Modern Deep Learning Methods
- A Learned Performance Model for Tensor Processing Units
- Optimizing Memory-Access Patterns for Deep Learning Accelerators
- Daydream: Accurately Estimating the Efficacy of Optimizations for DNN Training