Using Python for Model Inference in Deep Learning

Abstract

Python has become the de-facto language for training deep neural networks, coupling a large suite of scientific computing libraries with efficient libraries for tensor computation such as PyTorch or TensorFlow. However, when models are used for inference they are typically extracted from Python as TensorFlow graphs or TorchScript programs in order to meet performance and packaging constraints. The extraction process can be time consuming, impeding fast prototyping. We show how it is possible to meet these performance and packaging constraints while performing inference in Python. In particular, we present a way of using multiple Python interpreters within a single process to achieve scalable inference and describe a new container format for models that contains both native Python code and data. This approach simplifies the model deployment story by eliminating the model extraction step, and makes it easier to integrate existing performance-enhancing Python libraries. We evaluate our design on a suite of popular PyTorch models on Github, showing how they can be packaged in our inference format, and comparing their performance to TorchScript. For larger models, our packaged Python models perform the same as TorchScript, and for smaller models where there is some Python overhead, our multi-interpreter approach ensures inference is still scalable.

References in corpus (8)

• YOLOv3: An Incremental Improvement
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• Momentum Contrast for Unsupervised Visual Representation Learning
• Deep Learning Recommendation Model for Personalization and Recommendation Systems
• TVM: An Automated End-to-End Optimizing Compiler for Deep Learning
• Demucs: Deep Extractor for Music Sources with extra unlabeled data remixed
• Deep Slow Motion Video Reconstruction with Hybrid Imaging System
• Learning to Paint With Model-based Deep Reinforcement Learning