An Orchestrated Empirical Study on Deep Learning Frameworks and Platforms
arXiv:1811.05187
Abstract
Deep learning (DL) has recently achieved tremendous success in a variety of cutting-edge applications, e.g., image recognition, speech and natural language processing, and autonomous driving. Besides the available big data and hardware evolution, DL frameworks and platforms play a key role to catalyze the research, development, and deployment of DL intelligent solutions. However, the difference in computation paradigm, architecture design and implementation of existing DL frameworks and platforms brings challenges for DL software development, deployment, maintenance, and migration. Up to the present, it still lacks a comprehensive study on how current diverse DL frameworks and platforms influence the DL software development process. In this paper, we initiate the first step towards the investigation on how existing state-of-the-art DL frameworks (i.e., TensorFlow, Theano, and Torch) and platforms (i.e., server/desktop, web, and mobile) support the DL software development activities. We perform an in-depth and comparative evaluation on metrics such as learning accuracy, DL model size, robustness, and performance, on state-of-the-art DL frameworks across platforms using two popular datasets MNIST and CIFAR-10. Our study reveals that existing DL frameworks still suffer from compatibility issues, which becomes even more severe when it comes to different platforms. We pinpoint the current challenges and opportunities towards developing high quality and compatible DL systems. To ignite further investigation along this direction to address urgent industrial demands of intelligent solutions, we make all of our assembled feasible toolchain and dataset publicly available.
References in corpus (12)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
- Explaining and Harnessing Adversarial Examples
- Caffe: Convolutional Architecture for Fast Feature Embedding
- TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing
- Deep Learning with Dynamic Computation Graphs
- Benchmarking State-of-the-Art Deep Learning Software Tools
- Combinatorial Testing for Deep Learning Systems
- Benchmarking Usability and Performance of Multicore Languages
- DeepMutation: Mutation Testing of Deep Learning Systems
- Comparative Analysis of Open Source Frameworks for Machine Learning with Use Case in Single-Threaded and Multi-Threaded Modes
- Deep Learning Towards Mobile Applications