TBD: Benchmarking and Analyzing Deep Neural Network Training

Abstract

The recent popularity of deep neural networks (DNNs) has generated a lot of research interest in performing DNN-related computation efficiently. However, the primary focus is usually very narrow and limited to (i) inference -- i.e. how to efficiently execute already trained models and (ii) image classification networks as the primary benchmark for evaluation. Our primary goal in this work is to break this myopic view by (i) proposing a new benchmark for DNN training, called TBD (TBD is short for Training Benchmark for DNNs), that uses a representative set of DNN models that cover a wide range of machine learning applications: image classification, machine translation, speech recognition, object detection, adversarial networks, reinforcement learning, and (ii) by performing an extensive performance analysis of training these different applications on three major deep learning frameworks (TensorFlow, MXNet, CNTK) across different hardware configurations (single-GPU, multi-GPU, and multi-machine). TBD currently covers six major application domains and eight different state-of-the-art models. We present a new toolchain for performance analysis for these models that combines the targeted usage of existing performance analysis tools, careful selection of new and existing metrics and methodologies to analyze the results, and utilization of domain specific characteristics of DNN training. We also build a new set of tools for memory profiling in all three major frameworks; much needed tools that can finally shed some light on precisely how much memory is consumed by different data structures (weights, activations, gradients, workspace) in DNN training. By using our tools and methodologies, we make several important observations and recommendations on where the future research and optimization of DNN training should be focused.

References in corpus (18)

• Sequence to Sequence Learning with Neural Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• End to End Learning for Self-Driving Cars
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Improved Training of Wasserstein GANs
• DeepXplore: Automated Whitebox Testing of Deep Learning Systems
• cuDNN: Efficient Primitives for Deep Learning
• YOLO9000: Better, Faster, Stronger
• An Empirical Evaluation of Deep Learning on Highway Driving
• A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
• Sockeye: A Toolkit for Neural Machine Translation
• Fathom: Reference Workloads for Modern Deep Learning Methods
• Benchmarking State-of-the-Art Deep Learning Software Tools
• ImageNet Training in Minutes
• Maximizing CNN Accelerator Efficiency Through Resource Partitioning

Cited by in corpus (19)

• Parameter Hub: a Rack-Scale Parameter Server for Distributed Deep Neural Network Training
• On-Device Machine Learning: An Algorithms and Learning Theory Perspective
• On the Utility of Gradient Compression in Distributed Training Systems
• HPC AI500: The Methodology, Tools, Roofline Performance Models, and Metrics for Benchmarking HPC AI Systems
• Towards Performance Clarity of Edge Video Analytics
• The Design and Implementation of a Scalable DL Benchmarking Platform
• Tango: A Deep Neural Network Benchmark Suite for Various Accelerators
• Frustrated with Replicating Claims of a Shared Model? A Solution
• DeepOBS: A Deep Learning Optimizer Benchmark Suite
• Benchmarking the Performance and Energy Efficiency of AI Accelerators for AI Training
• Demystifying the MLPerf Benchmark Suite
• Echo: Compiler-based GPU Memory Footprint Reduction for LSTM RNN Training
• AIBench Scenario: Scenario-distilling AI Benchmarking
• AIBench Training: Balanced Industry-Standard AI Training Benchmarking
• AIPerf: Automated machine learning as an AI-HPC benchmark
• HPC AI500: A Benchmark Suite for HPC AI Systems
• AIBench: An Agile Domain-specific Benchmarking Methodology and an AI Benchmark Suite
• SplitPlace: Intelligent Placement of Split Neural Nets in Mobile Edge Environments
• Progressive Compressed Records: Taking a Byte out of Deep Learning Data