Deep Learning Inference in Facebook Data Centers: Characterization, Performance Optimizations and Hardware Implications

Abstract

The application of deep learning techniques resulted in remarkable improvement of machine learning models. In this paper provides detailed characterizations of deep learning models used in many Facebook social network services. We present computational characteristics of our models, describe high performance optimizations targeting existing systems, point out their limitations and make suggestions for the future general-purpose/accelerated inference hardware. Also, we highlight the need for better co-design of algorithms, numerics and computing platforms to address the challenges of workloads often run in data centers.

References in corpus (21)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Sequence to Sequence Learning with Neural Networks
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Convolutional Sequence to Sequence Learning
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Quantizing deep convolutional networks for efficient inference: A whitepaper
• Non-Autoregressive Neural Machine Translation
• Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond
• Rosetta: Large scale system for text detection and recognition in images
• Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions
• WRPN: Wide Reduced-Precision Networks
• Glow: Graph Lowering Compiler Techniques for Neural Networks
• Neural Machine Translation and Sequence-to-sequence Models: A Tutorial
• Training Deep AutoEncoders for Collaborative Filtering
• Deep & Cross Network for Ad Click Predictions
• Bandana: Using Non-volatile Memory for Storing Deep Learning Models
• Bit Fusion: Bit-Level Dynamically Composable Architecture for Accelerating Deep Neural Networks
• Controllable Abstractive Summarization
• Learning Structured Natural Language Representations for Semantic Parsing

Cited by in corpus (33)

• Hardware Acceleration of Sparse and Irregular Tensor Computations of ML Models: A Survey and Insights
• Deep Learning Training in Facebook Data Centers: Design of Scale-up and Scale-out Systems
• DeLTA: GPU Performance Model for Deep Learning Applications with In-depth Memory System Traffic Analysis
• Mixed Dimension Embeddings with Application to Memory-Efficient Recommendation Systems
• SPRING: A Sparsity-Aware Reduced-Precision Monolithic 3D CNN Accelerator Architecture for Training and Inference
• TensorDIMM: A Practical Near-Memory Processing Architecture for Embeddings and Tensor Operations in Deep Learning
• Differentiable Neural Input Search for Recommender Systems
• First-Generation Inference Accelerator Deployment at Facebook
• Demonstrating Analog Inference on the BrainScaleS-2 Mobile System
• Post-Training 4-bit Quantization on Embedding Tables
• Mixed-Precision Embedding Using a Cache
• The Architectural Implications of Facebook's DNN-based Personalized Recommendation
• Parity Models: A General Framework for Coding-Based Resilience in ML Inference
• SMAUG: End-to-End Full-Stack Simulation Infrastructure for Deep Learning Workloads
• Accelerating Bandwidth-Bound Deep Learning Inference with Main-Memory Accelerators
• Adaptive Dense-to-Sparse Paradigm for Pruning Online Recommendation System with Non-Stationary Data
• Time-based Sequence Model for Personalization and Recommendation Systems
• Centaur: A Chiplet-based, Hybrid Sparse-Dense Accelerator for Personalized Recommendations
• SMASH: Sparse Matrix Atomic Scratchpad Hashing
• Characterizing Deep Learning Training Workloads on Alibaba-PAI
• The OoO VLIW JIT Compiler for GPU Inference
• RecSSD: Near Data Processing for Solid State Drive Based Recommendation Inference
• Proximu$: Efficiently Scaling DNN Inference in Multi-core CPUs through Near-Cache Compute
• Optimizing Memory Placement using Evolutionary Graph Reinforcement Learning
• Improving Efficiency in Neural Network Accelerator Using Operands Hamming Distance optimization
• Semantically Constrained Memory Allocation (SCMA) for Embedding in Efficient Recommendation Systems
• DCAF: A Dynamic Computation Allocation Framework for Online Serving System
• Accelerating Recommender Systems via Hardware "scale-in"
• NeuMMU: Architectural Support for Efficient Address Translations in Neural Processing Units
• Production Ranking Systems: A Review
• JIZHI: A Fast and Cost-Effective Model-As-A-Service System for Web-Scale Online Inference at Baidu
• Low-Precision Hardware Architectures Meet Recommendation Model Inference at Scale
• High performance and energy efficient inference for deep learning on ARM processors