Mixed Dimension Embeddings with Application to Memory-Efficient Recommendation Systems

Abstract

Embedding representations power machine intelligence in many applications, including recommendation systems, but they are space intensive -- potentially occupying hundreds of gigabytes in large-scale settings. To help manage this outsized memory consumption, we explore mixed dimension embeddings, an embedding layer architecture in which a particular embedding vector's dimension scales with its query frequency. Through theoretical analysis and systematic experiments, we demonstrate that using mixed dimensions can drastically reduce the memory usage, while maintaining and even improving the ML performance. Empirically, we show that the proposed mixed dimension layers improve accuracy by 0.1% using half as many parameters or maintain it using 16X fewer parameters for click-through rate prediction task on the Criteo Kaggle dataset.

References in corpus (22)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• On the Convergence of Adam and Beyond
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems
• ProtVec: A Continuous Distributed Representation of Biological Sequences
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• A Simpler Approach to Matrix Completion
• DeepFM: A Factorization-Machine based Neural Network for CTR Prediction
• Restricted strong convexity and weighted matrix completion: Optimal bounds with noise
• Deep Learning Recommendation Model for Personalization and Recommendation Systems
• Guaranteed Matrix Completion via Non-convex Factorization
• Compression-aware Training of Deep Networks
• On the Dimensionality of Word Embedding
• Hierarchy-based Image Embeddings for Semantic Image Retrieval
• On the Difficulty of Evaluating Baselines: A Study on Recommender Systems
• Deep Learning Inference in Facebook Data Centers: Characterization, Performance Optimizations and Hardware Implications
• Adaptive Input Representations for Neural Language Modeling
• Tensorized Embedding Layers for Efficient Model Compression
• Compressing Word Embeddings via Deep Compositional Code Learning
• On Periodic Functions as Regularizers for Quantization of Neural Networks
• On the Dimensionality of Embeddings for Sparse Features and Data
• Saec: Similarity-Aware Embedding Compression in Recommendation Systems

Cited by in corpus (14)

• DeepRecSys: A System for Optimizing End-To-End At-scale Neural Recommendation Inference
• AutoEmb: Automated Embedding Dimensionality Search in Streaming Recommendations
• Learnable Embedding Sizes for Recommender Systems
• Differentiable Neural Input Search for Recommender Systems
• Memory-efficient Embedding for Recommendations
• Mixed-Precision Embedding Using a Cache
• Understanding Training Efficiency of Deep Learning Recommendation Models at Scale
• AutoLoss: Automated Loss Function Search in Recommendations
• Anchor & Transform: Learning Sparse Embeddings for Large Vocabularies
• FINT: Field-aware INTeraction Neural Network For CTR Prediction
• Random Offset Block Embedding Array (ROBE) for CriteoTB Benchmark MLPerf DLRM Model : 1000$\times$ Compression and 3.1$\times$ Faster Inference
• Semantically Constrained Memory Allocation (SCMA) for Embedding in Efficient Recommendation Systems
• Learning Effective and Efficient Embedding via an Adaptively-Masked Twins-based Layer
• ECRM: Efficient Fault Tolerance for Recommendation Model Training via Erasure Coding