Comparing the costs of abstraction for DL frameworks
arXiv:2012.07163
Abstract
High level abstractions for implementing, training, and testing Deep Learning (DL) models abound. Such frameworks function primarily by abstracting away the implementation details of arbitrary neural architectures, thereby enabling researchers and engineers to focus on design. In principle, such frameworks could be "zero-cost abstractions"; in practice, they incur translation and indirection overheads. We study at which points exactly in the engineering life-cycle of a DL model the highest costs are paid and whether they can be mitigated. We train, test, and evaluate a representative DL model using PyTorch, LibTorch, TorchScript, and cuDNN on representative datasets, comparing accuracy, execution time and memory efficiency.
References in corpus (12)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
- Deep Residual Learning for Image Recognition
- Language Models are Few-Shot Learners
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
- Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
- Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions
- MLIR: A Compiler Infrastructure for the End of Moore's Law
- Fixup Initialization: Residual Learning Without Normalization
- TFLMS: Large Model Support in TensorFlow by Graph Rewriting
- Stripe: Tensor Compilation via the Nested Polyhedral Model