HET: Scaling out Huge Embedding Model Training via Cache-enabled Distributed Framework

Abstract

Embedding models have been an effective learning paradigm for high-dimensional data. However, one open issue of embedding models is that their representations (latent factors) often result in large parameter space. We observe that existing distributed training frameworks face a scalability issue of embedding models since updating and retrieving the shared embedding parameters from servers usually dominates the training cycle. In this paper, we propose HET, a new system framework that significantly improves the scalability of huge embedding model training. We embrace skewed popularity distributions of embeddings as a performance opportunity and leverage it to address the communication bottleneck with an embedding cache. To ensure consistency across the caches, we incorporate a new consistency model into HET design, which provides fine-grained consistency guarantees on a per-embedding basis. Compared to previous work that only allows staleness for read operations, HET also utilizes staleness for write operations. Evaluations on six representative tasks show that HET achieves up to 88% embedding communication reductions and up to 20.68x performance speedup over the state-of-the-art baselines.

References in corpus (15)

• Efficient Estimation of Word Representations in Vector Space
• DeepWalk: Online Learning of Social Representations
• TensorFlow: A system for large-scale machine learning
• Neural Graph Collaborative Filtering
• Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks
• xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems
• DeepFM: A Factorization-Machine based Neural Network for CTR Prediction
• Deep Learning Recommendation Model for Personalization and Recommendation Systems
• Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization
• Document Embedding with Paragraph Vectors
• PyTorch Distributed: Experiences on Accelerating Data Parallel Training
• Distributed Hierarchical GPU Parameter Server for Massive Scale Deep Learning Ads Systems
• Asynchronous Decentralized Parallel Stochastic Gradient Descent
• Parallax: Sparsity-aware Data Parallel Training of Deep Neural Networks
• Asynchronous Training of Word Embeddings for Large Text Corpora

Cited by in corpus (8)

• Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism
• BlindFL: Vertical Federated Machine Learning without Peeking into Your Data
• FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement
• Towards Communication-efficient Vertical Federated Learning Training via Cache-enabled Local Updates
• Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving
• A Comprehensive Benchmark on Spectral GNNs: The Impact on Efficiency, Memory, and Effectiveness
• MEMO: Fine-grained Tensor Management For Ultra-long Context LLM Training
• Good Intentions: Adaptive Parameter Management via Intent Signaling