Distributed Hierarchical GPU Parameter Server for Massive Scale Deep Learning Ads Systems

Abstract

Neural networks of ads systems usually take input from multiple resources, e.g., query-ad relevance, ad features and user portraits. These inputs are encoded into one-hot or multi-hot binary features, with typically only a tiny fraction of nonzero feature values per example. Deep learning models in online advertising industries can have terabyte-scale parameters that do not fit in the GPU memory nor the CPU main memory on a computing node. For example, a sponsored online advertising system can contain more than $10^{11}$ sparse features, making the neural network a massive model with around 10 TB parameters. In this paper, we introduce a distributed GPU hierarchical parameter server for massive scale deep learning ads systems. We propose a hierarchical workflow that utilizes GPU High-Bandwidth Memory, CPU main memory and SSD as 3-layer hierarchical storage. All the neural network training computations are contained in GPUs. Extensive experiments on real-world data confirm the effectiveness and the scalability of the proposed system. A 4-node hierarchical GPU parameter server can train a model more than 2X faster than a 150-node in-memory distributed parameter server in an MPI cluster. In addition, the price-performance ratio of our proposed system is 4-9 times better than an MPI-cluster solution.

References in corpus (3)

• Hashing Algorithms for Large-Scale Learning
• Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusters
• Bandana: Using Non-volatile Memory for Storing Deep Learning Models

Cited by in corpus (21)

• Sustainable AI: Environmental Implications, Challenges and Opportunities
• HET: Scaling out Huge Embedding Model Training via Cache-enabled Distributed Framework
• ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning
• Accelerating Recommendation System Training by Leveraging Popular Choices
• Merlin HugeCTR: GPU-accelerated Recommender System Training and Inference
• A GPU-specialized Inference Parameter Server for Large-Scale Deep Recommendation Models
• Reformulating CTR Prediction: Learning Invariant Feature Interactions for Recommendation
• Mixed-Precision Embedding Using a Cache
• Taming Momentum in a Distributed Asynchronous Environment
• TT-Rec: Tensor Train Compression for Deep Learning Recommendation Models
• User Response Prediction in Online Advertising
• Understanding Training Efficiency of Deep Learning Recommendation Models at Scale
• HeterPS: Distributed Deep Learning With Reinforcement Learning Based Scheduling in Heterogeneous Environments
• RecSSD: Near Data Processing for Solid State Drive Based Recommendation Inference
• CPR: Understanding and Improving Failure Tolerant Training for Deep Learning Recommendation with Partial Recovery
• Efficient Soft-Error Detection for Low-precision Deep Learning Recommendation Models
• Good Intentions: Adaptive Parameter Management via Intent Signaling
• ScaleFreeCTR: MixCache-based Distributed Training System for CTR Models with Huge Embedding Table
• WeiPS: a symmetric fusion model framework for large-scale online learning
• Low-Precision Hardware Architectures Meet Recommendation Model Inference at Scale
• Supporting Massive DLRM Inference Through Software Defined Memory