Benchmarking State-of-the-Art Deep Learning Software Tools

Abstract

Deep learning has been shown as a successful machine learning method for a variety of tasks, and its popularity results in numerous open-source deep learning software tools. Training a deep network is usually a very time-consuming process. To address the computational challenge in deep learning, many tools exploit hardware features such as multi-core CPUs and many-core GPUs to shorten the training time. However, different tools exhibit different features and running performance when training different types of deep networks on different hardware platforms, which makes it difficult for end users to select an appropriate pair of software and hardware. In this paper, we aim to make a comparative study of the state-of-the-art GPU-accelerated deep learning software tools, including Caffe, CNTK, MXNet, TensorFlow, and Torch. We first benchmark the running performance of these tools with three popular types of neural networks on two CPU platforms and three GPU platforms. We then benchmark some distributed versions on multiple GPUs. Our contribution is two-fold. First, for end users of deep learning tools, our benchmarking results can serve as a guide to selecting appropriate hardware platforms and software tools. Second, for software developers of deep learning tools, our in-depth analysis points out possible future directions to further optimize the running performance.

Revision history: 1. Revise ResNet-50 configuration in MXNet. 2. Add faster implementation of ResNet-56 in TensorFlow with multiple GPUs

References in corpus (9)

• Deep Learning in Neural Networks: An Overview
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Recurrent Neural Network Regularization
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• cuDNN: Efficient Primitives for Deep Learning
• Revisiting Distributed Synchronous SGD
• Distributed Deep Learning Using Synchronous Stochastic Gradient Descent
• Comparative Study of Deep Learning Software Frameworks
• The Potential of the Intel Xeon Phi for Supervised Deep Learning

Cited by in corpus (19)

• Parameter Hub: a Rack-Scale Parameter Server for Distributed Deep Neural Network Training
• Neural Models for Information Retrieval
• Characterizing Deep-Learning I/O Workloads in TensorFlow
• TBD: Benchmarking and Analyzing Deep Neural Network Training
• Speeding up Convolutional Neural Networks By Exploiting the Sparsity of Rectifier Units
• A Survey on Deep Learning Toolkits and Libraries for Intelligent User Interfaces
• Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs
• A detailed comparative study of open source deep learning frameworks
• Distributed Training Large-Scale Deep Architectures
• Performance Evaluation of Deep Learning Tools in Docker Containers
• A DAG Model of Synchronous Stochastic Gradient Descent in Distributed Deep Learning
• Barista - a Graphical Tool for Designing and Training Deep Neural Networks
• Frustrated with Replicating Claims of a Shared Model? A Solution
• GOAT: GPU Outsourcing of Deep Learning Training With Asynchronous Probabilistic Integrity Verification Inside Trusted Execution Environment
• GPGPU Performance Estimation with Core and Memory Frequency Scaling
• An Orchestrated Empirical Study on Deep Learning Frameworks and Platforms
• Efficient logic architecture in training gradient boosting decision tree for high-performance and edge computing
• Improving the network traffic classification using the Packet Vision approach
• Distributed deep learning on edge-devices: feasibility via adaptive compression