Comparative Study of Deep Learning Software Frameworks

Abstract

Deep learning methods have resulted in significant performance improvements in several application domains and as such several software frameworks have been developed to facilitate their implementation. This paper presents a comparative study of five deep learning frameworks, namely Caffe, Neon, TensorFlow, Theano, and Torch, on three aspects: extensibility, hardware utilization, and speed. The study is performed on several types of deep learning architectures and we evaluate the performance of the above frameworks when employed on a single machine for both (multi-threaded) CPU and GPU (Nvidia Titan X) settings. The speed performance metrics used here include the gradient computation time, which is important during the training phase of deep networks, and the forward time, which is important from the deployment perspective of trained networks. For convolutional networks, we also report how each of these frameworks support various convolutional algorithms and their corresponding performance. From our experiments, we observe that Theano and Torch are the most easily extensible frameworks. We observe that Torch is best suited for any deep architecture on CPU, followed by Theano. It also achieves the best performance on the GPU for large convolutional and fully connected networks, followed closely by Neon. Theano achieves the best performance on GPU for training and deployment of LSTM networks. Caffe is the easiest for evaluating the performance of standard deep architectures. Finally, TensorFlow is a very flexible framework, similar to Theano, but its performance is currently not competitive compared to the other studied frameworks.

Submitted to KDD 2016 with TensorFlow results added. At the time of submission to KDD, TensorFlow was available only with cuDNN v.2 and thus its performance is reported with that version

References in corpus (5)

• Natural Language Processing (almost) from Scratch
• Theano: new features and speed improvements
• Fast Training of Convolutional Networks through FFTs
• Fast Convolutional Nets With fbfft: A GPU Performance Evaluation
• Theano-based Large-Scale Visual Recognition with Multiple GPUs

Cited by in corpus (17)

• Predicting Process Behaviour using Deep Learning
• Benchmarking State-of-the-Art Deep Learning Software Tools
• Deep Learning vs. Traditional Computer Vision
• pCAMP: Performance Comparison of Machine Learning Packages on the Edges
• A Survey on Deep Learning Methods for Robot Vision
• Deep Learning in Bioinformatics
• Characterizing the Deep Neural Networks Inference Performance of Mobile Applications
• Efficient Memory Management for GPU-based Deep Learning Systems
• Deep Learning in Mining Biological Data
• Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs
• An Empirical Study towards Characterizing Deep Learning Development and Deployment across Different Frameworks and Platforms
• A detailed comparative study of open source deep learning frameworks
• Distributed Training Large-Scale Deep Architectures
• A DAG Model of Synchronous Stochastic Gradient Descent in Distributed Deep Learning
• Server, server in the cloud. Who is the fairest in the crowd?
• Distributed Machine Learning in Materials that Couple Sensing, Actuation, Computation and Communication
• Cloud-based or On-device: An Empirical Study of Mobile Deep Inference