Daydream: Accurately Estimating the Efficacy of Optimizations for DNN Training

Abstract

Modern deep neural network (DNN) training jobs use complex and heterogeneous software/hardware stacks. The efficacy of software-level optimizations can vary significantly when used in different deployment configurations. It is onerous and error-prone for ML practitioners and system developers to implement each optimization separately, and determine which ones will improve performance in their own configurations. Unfortunately, existing profiling tools do not aim to answer predictive questions such as "How will optimization X affect the performance of my model?". We address this critical limitation, and proposes a new profiling tool, Daydream, to help programmers efficiently explore the efficacy of DNN optimizations. Daydream models DNN execution with a fine-grained dependency graph based on low-level traces collected by CUPTI, and predicts runtime by simulating execution based on the dependency graph. Daydream maps the low-level traces using DNN domain-specific knowledge, and introduces a set of graph-transformation primitives that can easily model a wide variety of optimizations. We show that Daydream is able to model most mainstream DNN optimization techniques, and accurately predict the efficacy of optimizations that will result in significant performance improvements.

References in corpus (17)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• cuDNN: Efficient Primitives for Deep Learning
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• Mixed Precision Training
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• Training Deep Nets with Sublinear Memory Cost
• Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes
• Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions
• TVM: An Automated End-to-End Optimizing Compiler for Deep Learning
• Mixed Precision Training of Convolutional Neural Networks using Integer Operations
• Restructuring Batch Normalization to Accelerate CNN Training
• Priority-based Parameter Propagation for Distributed DNN Training
• Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGD

Cited by in corpus (3)

• On the Utility of Gradient Compression in Distributed Training Systems
• A Runtime-Based Computational Performance Predictor for Deep Neural Network Training
• DistIR: An Intermediate Representation and Simulator for Efficient Neural Network Distribution