Real-Time Machine Learning: The Missing Pieces

Abstract

Machine learning applications are increasingly deployed not only to serve predictions using static models, but also as tightly-integrated components of feedback loops involving dynamic, real-time decision making. These applications pose a new set of requirements, none of which are difficult to achieve in isolation, but the combination of which creates a challenge for existing distributed execution frameworks: computation with millisecond latency at high throughput, adaptive construction of arbitrary task graphs, and execution of heterogeneous kernels over diverse sets of resources. We assert that a new distributed execution framework is needed for such ML applications and propose a candidate approach with a proof-of-concept architecture that achieves a 63x performance improvement over a state-of-the-art execution framework for a representative application.

arXiv comment6 pages, 3 figures

References in corpus (6)

• TensorFlow: A system for large-scale machine learning
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Massively Parallel Methods for Deep Reinforcement Learning
• The Missing Piece in Complex Analytics: Low Latency, Scalable Model Management and Serving with Velox

Cited by in corpus (4)

• Ray: A Distributed Framework for Emerging AI Applications
• Time-sensitive Learning for Heterogeneous Federated Edge Intelligence
• PASS-GLM: polynomial approximate sufficient statistics for scalable Bayesian GLM inference
• Real-time policy generation and its application to robot grasping