DeepRecSys: A System for Optimizing End-To-End At-scale Neural Recommendation Inference

Abstract

Neural personalized recommendation is the corner-stone of a wide collection of cloud services and products, constituting significant compute demand of the cloud infrastructure. Thus, improving the execution efficiency of neural recommendation directly translates into infrastructure capacity saving. In this paper, we devise a novel end-to-end modeling infrastructure, DeepRecInfra, that adopts an algorithm and system co-design methodology to custom-design systems for recommendation use cases. Leveraging the insights from the recommendation characterization, a new dynamic scheduler, DeepRecSched, is proposed to maximize latency-bounded throughput by taking into account characteristics of inference query size and arrival patterns, recommendation model architectures, and underlying hardware systems. By doing so, system throughput is doubled across the eight industry-representative recommendation models. Finally, design, deployment, and evaluation in at-scale production datacenter shows over 30% latency reduction across a wide variety of recommendation models running on hundreds of machines.

References in corpus (9)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Deep Residual Learning for Image Recognition
• Deep Learning Recommendation Model for Personalization and Recommendation Systems
• Benchmarking TPU, GPU, and CPU Platforms for Deep Learning
• Fathom: Reference Workloads for Modern Deep Learning Methods
• MLPerf Training Benchmark
• Compositional Embeddings Using Complementary Partitions for Memory-Efficient Recommendation Systems
• Mixed Dimension Embeddings with Application to Memory-Efficient Recommendation Systems
• The Architectural Implications of Facebook's DNN-based Personalized Recommendation

Cited by in corpus (13)

• First-Generation Inference Accelerator Deployment at Facebook
• TT-Rec: Tensor Train Compression for Deep Learning Recommendation Models
• Faa$T: A Transparent Auto-Scaling Cache for Serverless Applications
• MicroRec: Efficient Recommendation Inference by Hardware and Data Structure Solutions
• Accelerating Bandwidth-Bound Deep Learning Inference with Main-Memory Accelerators
• Understanding Training Efficiency of Deep Learning Recommendation Models at Scale
• RecSSD: Near Data Processing for Solid State Drive Based Recommendation Inference
• INFaaS: A Model-less and Managed Inference Serving System
• LazyDP: Co-Designing Algorithm-Software for Scalable Training of Differentially Private Recommendation Models
• Accelerating Recommender Systems via Hardware "scale-in"
• ECRM: Efficient Fault Tolerance for Recommendation Model Training via Erasure Coding
• Alternate Model Growth and Pruning for Efficient Training of Recommendation Systems
• Low-Precision Hardware Architectures Meet Recommendation Model Inference at Scale