TensorFlow-Serving: Flexible, High-Performance ML Serving

Abstract

We describe TensorFlow-Serving, a system to serve machine learning models inside Google which is also available in the cloud and via open-source. It is extremely flexible in terms of the types of ML platforms it supports, and ways to integrate with systems that convey new models and updated versions from training to serving. At the same time, the core code paths around model lookup and inference have been carefully optimized to avoid performance pitfalls observed in naive implementations. Google uses it in many production deployments, including a multi-tenant model hosting service called TFS^2.

Presented at NIPS 2017 Workshop on ML Systems (http://learningsys.org/nips17/acceptedpapers.html)

References in corpus (2)

• The Missing Piece in Complex Analytics: Low Latency, Scalable Model Management and Serving with Velox
• On-the-fly Operation Batching in Dynamic Computation Graphs

Cited by in corpus (15)

• KAISA: An Adaptive Second-Order Optimizer Framework for Deep Neural Networks
• ModelHub.AI: Dissemination Platform for Deep Learning Models
• Multi-model Machine Learning Inference Serving with GPU Spatial Partitioning
• InferBench: Understanding Deep Learning Inference Serving with an Automatic Benchmarking System
• ModelCI-e: Enabling Continual Learning in Deep Learning Serving Systems
• HOLMES: Health OnLine Model Ensemble Serving for Deep Learning Models in Intensive Care Units
• Towards Personalized and Semantic Retrieval: An End-to-End Solution for E-commerce Search via Embedding Learning
• CardioLearn: A Cloud Deep Learning Service for Cardiac Disease Detection from Electrocardiogram
• No DNN Left Behind: Improving Inference in the Cloud with Multi-Tenancy
• Cocktail: Leveraging Ensemble Learning for Optimized Model Serving in Public Cloud
• JIZHI: A Fast and Cost-Effective Model-As-A-Service System for Web-Scale Online Inference at Baidu
• AIBench: An Agile Domain-specific Benchmarking Methodology and an AI Benchmark Suite
• Integration of TensorFlow based Acoustic Model with Kaldi WFST Decoder
• FlexServe: Deployment of PyTorch Models as Flexible REST Endpoints
• PACSET (Packed Serialized Trees): Reducing Inference Latency for Tree Ensemble Deployment