Clipper: A Low-Latency Online Prediction Serving System

Abstract

Machine learning is being deployed in a growing number of applications which demand real-time, accurate, and robust predictions under heavy query load. However, most machine learning frameworks and systems only address model training and not deployment. In this paper, we introduce Clipper, a general-purpose low-latency prediction serving system. Interposing between end-user applications and a wide range of machine learning frameworks, Clipper introduces a modular architecture to simplify model deployment across frameworks and applications. Furthermore, by introducing caching, batching, and adaptive model selection techniques, Clipper reduces prediction latency and improves prediction throughput, accuracy, and robustness without modifying the underlying machine learning frameworks. We evaluate Clipper on four common machine learning benchmark datasets and demonstrate its ability to meet the latency, accuracy, and throughput demands of online serving applications. Finally, we compare Clipper to the TensorFlow Serving system and demonstrate that we are able to achieve comparable throughput and latency while enabling model composition and online learning to improve accuracy and render more robust predictions.

References in corpus (6)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Caffe: Convolutional Architecture for Fast Feature Embedding
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• The Missing Piece in Complex Analytics: Low Latency, Scalable Model Management and Serving with Velox
• Making Contextual Decisions with Low Technical Debt
• Large Scale Language Modeling in Automatic Speech Recognition

Cited by in corpus (11)

• Clover: Toward Sustainable AI with Carbon-Aware Machine Learning Inference Service
• Software Engineering for AI-Based Systems: A Survey
• BlinkML: Efficient Maximum Likelihood Estimation with Probabilistic Guarantees
• NeurDB: An AI-powered Autonomous Data System
• Fusion: Efficient and Secure Inference Resilient to Malicious Servers
• TrIMS: Transparent and Isolated Model Sharing for Low Latency Deep LearningInference in Function as a Service Environments
• RIBBON: Cost-Effective and QoS-Aware Deep Learning Model Inference using a Diverse Pool of Cloud Computing Instances
• KAIROS: Building Cost-Efficient Machine Learning Inference Systems with Heterogeneous Cloud Resources
• Teola: Towards End-to-End Optimization of LLM-based Applications
• HOLMES: Health OnLine Model Ensemble Serving for Deep Learning Models in Intensive Care Units
• Net2Vec: Deep Learning for the Network