TensorFlow-Serving: Flexible, High-Performance ML Serving

Abstract

We describe TensorFlow-Serving, a system to serve machine learning models inside Google which is also available in the cloud and via open-source. It is extremely flexible in terms of the types of ML platforms it supports, and ways to integrate with systems that convey new models and updated versions from training to serving. At the same time, the core code paths around model lookup and inference have been carefully optimized to avoid performance pitfalls observed in naive implementations. Google uses it in many production deployments, including a multi-tenant model hosting service called TFS^2.

Presented at NIPS 2017 Workshop on ML Systems (http://learningsys.org/nips17/acceptedpapers.html)

References in corpus (2)

• The Missing Piece in Complex Analytics: Low Latency, Scalable Model Management and Serving with Velox
• On-the-fly Operation Batching in Dynamic Computation Graphs

Cited by in corpus (32)

• NeuPIMs: NPU-PIM Heterogeneous Acceleration for Batched LLM Inferencing
• TF-Ranking: Scalable TensorFlow Library for Learning-to-Rank
• AIBench: An Industry Standard Internet Service AI Benchmark Suite
• IDK Cascades: Fast Deep Learning by Learning not to Overthink
• Cloudy with high chance of DBMS: A 10-year prediction for Enterprise-Grade ML
• Characterizing the Deep Neural Networks Inference Performance of Mobile Applications
• KAISA: An Adaptive Second-Order Optimizer Framework for Deep Neural Networks
• Multi-model Machine Learning Inference Serving with GPU Spatial Partitioning
• ModelHub.AI: Dissemination Platform for Deep Learning Models
• The Design and Implementation of a Scalable DL Benchmarking Platform
• Large-Scale Intelligent Microservices
• Frustrated with Replicating Claims of a Shared Model? A Solution
• InferBench: Understanding Deep Learning Inference Serving with an Automatic Benchmarking System
• DeepRT: A Soft Real Time Scheduler for Computer Vision Applications on the Edge
• ModelCI-e: Enabling Continual Learning in Deep Learning Serving Systems
• Towards ML Engineering: A Brief History Of TensorFlow Extended (TFX)
• Towards Personalized and Semantic Retrieval: An End-to-End Solution for E-commerce Search via Embedding Learning
• HOLMES: Health OnLine Model Ensemble Serving for Deep Learning Models in Intensive Care Units
• CardioLearn: A Cloud Deep Learning Service for Cardiac Disease Detection from Electrocardiogram
• AIBench Scenario: Scenario-distilling AI Benchmarking
• No DNN Left Behind: Improving Inference in the Cloud with Multi-Tenancy
• Cocktail: Leveraging Ensemble Learning for Optimized Model Serving in Public Cloud
• Packaging and Sharing Machine Learning Models via the Acumos AI Open Platform
• ModiPick: SLA-aware Accuracy Optimization For Mobile Deep Inference
• JIZHI: A Fast and Cost-Effective Model-As-A-Service System for Web-Scale Online Inference at Baidu
• Towards Inference Delivery Networks: Distributing Machine Learning with Optimality Guarantees
• Golden Grain: Building a Secure and Decentralized Model Marketplace for MLaaS
• AIBench: An Agile Domain-specific Benchmarking Methodology and an AI Benchmark Suite
• Integration of TensorFlow based Acoustic Model with Kaldi WFST Decoder
• FlexServe: Deployment of PyTorch Models as Flexible REST Endpoints
• PACSET (Packed Serialized Trees): Reducing Inference Latency for Tree Ensemble Deployment
• Hoplite: Efficient and Fault-Tolerant Collective Communication for Task-Based Distributed Systems