Compositional Embeddings Using Complementary Partitions for Memory-Efficient Recommendation Systems

Abstract

Modern deep learning-based recommendation systems exploit hundreds to thousands of different categorical features, each with millions of different categories ranging from clicks to posts. To respect the natural diversity within the categorical data, embeddings map each category to a unique dense representation within an embedded space. Since each categorical feature could take on as many as tens of millions of different possible categories, the embedding tables form the primary memory bottleneck during both training and inference. We propose a novel approach for reducing the embedding size in an end-to-end fashion by exploiting complementary partitions of the category set to produce a unique embedding vector for each category without explicit definition. By storing multiple smaller embedding tables based on each complementary partition and combining embeddings from each table, we define a unique embedding for each category at smaller memory cost. This approach may be interpreted as using a specific fixed codebook to ensure uniqueness of each category's representation. Our experimental results demonstrate the effectiveness of our approach over the hashing trick for reducing the size of the embedding tables in terms of model loss and accuracy, while retaining a similar reduction in the number of parameters.

11 pages, 7 figures, 1 table

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• On the Convergence of Adam and Beyond
• xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems
• DeepFM: A Factorization-Machine based Neural Network for CTR Prediction
• Deep Learning Recommendation Model for Personalization and Recommendation Systems
• Learning Tree-based Deep Model for Recommender Systems
• On the Dimensionality of Word Embedding
• Deep Interest Evolution Network for Click-Through Rate Prediction
• Tensorized Embedding Layers for Efficient Model Compression
• Learning K-way D-dimensional Discrete Codes for Compact Embedding Representations
• Deep Interest Network for Click-Through Rate Prediction
• Compressing Word Embeddings via Deep Compositional Code Learning
• Feature Hashing for Large Scale Multitask Learning
• On the Dimensionality of Embeddings for Sparse Features and Data
• Learning K-way D-dimensional Discrete Code For Compact Embedding Representations
• MARS: Memory Attention-Aware Recommender System

Cited by in corpus (21)

• Click-Through Rate Prediction in Online Advertising: A Literature Review
• Accelerating Recommendation System Training by Leveraging Popular Choices
• DeepRecSys: A System for Optimizing End-To-End At-scale Neural Recommendation Inference
• OptEmbed: Learning Optimal Embedding Table for Click-through Rate Prediction
• Embedding Compression in Recommender Systems: A Survey
• Learnable Embedding Sizes for Recommender Systems
• Adaptive Low-Precision Training for Embeddings in Click-Through Rate Prediction
• RecJPQ: Training Large-Catalogue Sequential Recommenders
• A Thorough Performance Benchmarking on Lightweight Embedding-based Recommender Systems
• Semantic IDs for Music Recommendation
• Learning to Embed Categorical Features without Embedding Tables for Recommendation
• Random Offset Block Embedding Array (ROBE) for CriteoTB Benchmark MLPerf DLRM Model : 1000$\times$ Compression and 3.1$\times$ Faster Inference
• Differentiable NAS Framework and Application to Ads CTR Prediction
• Semantically Constrained Memory Allocation (SCMA) for Embedding in Efficient Recommendation Systems
• Learning Effective and Efficient Embedding via an Adaptively-Masked Twins-based Layer
• Learning Elastic Embeddings for Customizing On-Device Recommenders
• Hessian-aware Quantized Node Embeddings for Recommendation
• Toward a benchmark for CTR prediction in online advertising: datasets, evaluation protocols and perspectives
• A Survey on Green Deep Learning
• METEOR: Learning Memory and Time Efficient Representations from Multi-modal Data Streams
• Coarse-to-Fine Lightweight Meta-Embedding for ID-Based Recommendation