Deep Learning Recommendation Model for Personalization and Recommendation Systems

Abstract

With the advent of deep learning, neural network-based recommendation models have emerged as an important tool for tackling personalization and recommendation tasks. These networks differ significantly from other deep learning networks due to their need to handle categorical features and are not well studied or understood. In this paper, we develop a state-of-the-art deep learning recommendation model (DLRM) and provide its implementation in both PyTorch and Caffe2 frameworks. In addition, we design a specialized parallelization scheme utilizing model parallelism on the embedding tables to mitigate memory constraints while exploiting data parallelism to scale-out compute from the fully-connected layers. We compare DLRM against existing recommendation models and characterize its performance on the Big Basin AI platform, demonstrating its usefulness as a benchmark for future algorithmic experimentation and system co-design.

10 pages, 6 figures

References in corpus (3)

• xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems
• DeepFM: A Factorization-Machine based Neural Network for CTR Prediction
• On the Dimensionality of Embeddings for Sparse Features and Data

Cited by in corpus (104)

• Sustainable AI: Environmental Implications, Challenges and Opportunities
• Compute Trends Across Three Eras of Machine Learning
• MLPerf Training Benchmark
• DGCN: Diversified Recommendation with Graph Convolutional Networks
• Hardware Acceleration of Sparse and Irregular Tensor Computations of ML Models: A Survey and Insights
• Compositional Embeddings Using Complementary Partitions for Memory-Efficient Recommendation Systems
• SMASH: Co-designing Software Compression and Hardware-Accelerated Indexing for Efficient Sparse Matrix Operations
• Understanding Data Storage and Ingestion for Large-Scale Deep Recommendation Model Training
• A Study of BFLOAT16 for Deep Learning Training
• HET: Scaling out Huge Embedding Model Training via Cache-enabled Distributed Framework
• BlindFL: Vertical Federated Machine Learning without Peeking into Your Data
• Deep Learning Training in Facebook Data Centers: Design of Scale-up and Scale-out Systems
• Accelerating Recommendation System Training by Leveraging Popular Choices
• MetaBalance: Improving Multi-Task Recommendations via Adapting Gradient Magnitudes of Auxiliary Tasks
• The OARF Benchmark Suite: Characterization and Implications for Federated Learning Systems
• Enabling Compute-Communication Overlap in Distributed Deep Learning Training Platforms
• DeepRecSys: A System for Optimizing End-To-End At-scale Neural Recommendation Inference
• Graph Neural Network contextual embedding for Deep Learning on Tabular Data
• Merlin HugeCTR: GPU-accelerated Recommender System Training and Inference
• OptEmbed: Learning Optimal Embedding Table for Click-through Rate Prediction
• Mixed Dimension Embeddings with Application to Memory-Efficient Recommendation Systems
• Scalable Second Order Optimization for Deep Learning
• Self-supervised Learning for Large-scale Item Recommendations
• Contemporary Recommendation Systems on Big Data and Their Applications: A Survey
• The Bouncer Problem: Challenges to Remote Explainability
• Developing a Recommendation Benchmark for MLPerf Training and Inference
• Bolt: Bridging the Gap between Auto-tuners and Hardware-native Performance
• A GPU-specialized Inference Parameter Server for Large-Scale Deep Recommendation Models
• First-Generation Inference Accelerator Deployment at Facebook
• Check-N-Run: A Checkpointing System for Training Deep Learning Recommendation Models
• Field-Programmable Gate Array Architecture for Deep Learning: Survey & Future Directions
• Disentangling User Interest and Conformity for Recommendation with Causal Embedding
• Victima: Drastically Increasing Address Translation Reach by Leveraging Underutilized Cache Resources
• Post-Training 4-bit Quantization on Embedding Tables
• Copernicus: Characterizing the Performance Implications of Compression Formats Used in Sparse Workloads
• Mixed-Precision Embedding Using a Cache
• Utopia: Fast and Efficient Address Translation via Hybrid Restrictive & Flexible Virtual-to-Physical Address Mappings
• The Architectural Implications of Facebook's DNN-based Personalized Recommendation
• TT-Rec: Tensor Train Compression for Deep Learning Recommendation Models
• RecJPQ: Training Large-Catalogue Sequential Recommenders
• MicroRec: Efficient Recommendation Inference by Hardware and Data Structure Solutions
• AutoDDL: Automatic Distributed Deep Learning with Near-Optimal Bandwidth Cost
• Exploring the limits of Concurrency in ML Training on Google TPUs
• User Response Prediction in Online Advertising
• Accelerating Bandwidth-Bound Deep Learning Inference with Main-Memory Accelerators
• Understanding Training Efficiency of Deep Learning Recommendation Models at Scale
• Revisiting BFloat16 Training
• Matching Algorithms: Fundamentals, Applications and Challenges
• IPA: Inference Pipeline Adaptation to Achieve High Accuracy and Cost-Efficiency
• RecNMP: Accelerating Personalized Recommendation with Near-Memory Processing
• Privacy-Aware Recommender Systems Challenge on Twitter's Home Timeline
• Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models
• Heterogeneity-aware Cross-school Electives Recommendation: a Hybrid Federated Approach
• Time-based Sequence Model for Personalization and Recommendation Systems
• Tearing Down the Memory Wall
• Centaur: A Chiplet-based, Hybrid Sparse-Dense Accelerator for Personalized Recommendations
• Modyn: Data-Centric Machine Learning Pipeline Orchestration
• LIBRA: Enabling Workload-aware Multi-dimensional Network Topology Optimization for Distributed Training of Large AI Models
• Adaptive Dense-to-Sparse Paradigm for Pruning Online Recommendation System with Non-Stationary Data
• Strength from Weakness: Fast Learning Using Weak Supervision
• CoSA: Scheduling by Constrained Optimization for Spatial Accelerators
• Arithmetic-Intensity-Guided Fault Tolerance for Neural Network Inference on GPUs
• Training with Multi-Layer Embeddings for Model Reduction
• RecSSD: Near Data Processing for Solid State Drive Based Recommendation Inference
• ShadowSync: Performing Synchronization in the Background for Highly Scalable Distributed Training
• CPR: Understanding and Improving Failure Tolerant Training for Deep Learning Recommendation with Partial Recovery
• Tensor Processing Primitives: A Programming Abstraction for Efficiency and Portability in Deep Learning & HPC Workloads
• Handling Position Bias for Unbiased Learning to Rank in Hotels Search
• LazyDP: Co-Designing Algorithm-Software for Scalable Training of Differentially Private Recommendation Models
• AIBench Training: Balanced Industry-Standard AI Training Benchmarking
• WWW: What, When, Where to Compute-in-Memory
• AutoRec: An Automated Recommender System
• Managing ML Pipelines: Feature Stores and the Coming Wave of Embedding Ecosystems
• Random Offset Block Embedding Array (ROBE) for CriteoTB Benchmark MLPerf DLRM Model : 1000$\times$ Compression and 3.1$\times$ Faster Inference
• SOLAR: Sparse Orthogonal Learned and Random Embeddings
• AIRCHITECT: Learning Custom Architecture Design and Mapping Space
• Differentiable NAS Framework and Application to Ads CTR Prediction
• Semantically Constrained Memory Allocation (SCMA) for Embedding in Efficient Recommendation Systems
• Accelerating Recommender Systems via Hardware "scale-in"
• A Survey of Large-Scale Deep Learning Serving System Optimization: Challenges and Opportunities
• MP2: A Momentum Contrast Approach for Recommendation with Pointwise and Pairwise Learning
• Training With Data Dependent Dynamic Learning Rates
• NG+ : A Multi-Step Matrix-Product Natural Gradient Method for Deep Learning
• PairConnect: A Compute-Efficient MLP Alternative to Attention
• Leaf-FM: A Learnable Feature Generation Factorization Machine for Click-Through Rate Prediction
• 3U-EdgeAI: Ultra-Low Memory Training, Ultra-Low BitwidthQuantization, and Ultra-Low Latency Acceleration
• ECRM: Efficient Fault Tolerance for Recommendation Model Training via Erasure Coding
• Review Regularized Neural Collaborative Filtering
• Towards a Standardized Representation for Deep Learning Collective Algorithms
• Self-Adaptive Reconfigurable Arrays (SARA): Using ML to Assist Scaling GEMM Acceleration
• Union: A Unified HW-SW Co-Design Ecosystem in MLIR for Evaluating Tensor Operations on Spatial Accelerators
• Using Python for Model Inference in Deep Learning
• Adversarial Mixture Of Experts with Category Hierarchy Soft Constraint
• Zeroshot Listwise Learning to Rank Algorithm for Recommendation
• DimensionRank: Personal Neural Representations for Personalized General Search
• Alternate Model Growth and Pruning for Efficient Training of Recommendation Systems
• ScaleFreeCTR: MixCache-based Distributed Training System for CTR Models with Huge Embedding Table
• Low-Precision Hardware Architectures Meet Recommendation Model Inference at Scale
• Modeling Orders of User Behaviors via Differentiable Sorting: A Multi-task Framework to Predicting User Post-click Conversion
• Suggest, Complement, Inspire: Story of Two Tower Recommendations at Allegro.com
• LogitMat : Zeroshot Learning Algorithm for Recommender Systems without Transfer Learning or Pretrained Models
• Parameterized Explanations for Investor / Company Matching
• Supporting Massive DLRM Inference Through Software Defined Memory
• RASA: Efficient Register-Aware Systolic Array Matrix Engine for CPU