Memory-efficient Embedding for Recommendations

Abstract

Practical large-scale recommender systems usually contain thousands of feature fields from users, items, contextual information, and their interactions. Most of them empirically allocate a unified dimension to all feature fields, which is memory inefficient. Thus it is highly desired to assign different embedding dimensions to different feature fields according to their importance and predictability. Due to the large amounts of feature fields and the nuanced relationship between embedding dimensions with feature distributions and neural network architectures, manually allocating embedding dimensions in practical recommender systems can be very difficult. To this end, we propose an AutoML based framework (AutoDim) in this paper, which can automatically select dimensions for different feature fields in a data-driven fashion. Specifically, we first proposed an end-to-end differentiable framework that can calculate the weights over various dimensions for feature fields in a soft and continuous manner with an AutoML based optimization algorithm; then we derive a hard and discrete embedding component architecture according to the maximal weights and retrain the whole recommender framework. We conduct extensive experiments on benchmark datasets to validate the effectiveness of the AutoDim framework.

References in corpus (16)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Empirical Analysis of Predictive Algorithms for Collaborative Filtering
• Neural Architecture Search with Reinforcement Learning
• Categorical Reparameterization with Gumbel-Softmax
• xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems
• Efficient Neural Architecture Search via Parameter Sharing
• Session-based Recommendations with Recurrent Neural Networks
• Neural Architecture Optimization
• SMASH: One-Shot Model Architecture Search through HyperNetworks
• SNAS: Stochastic Neural Architecture Search
• Path-Level Network Transformation for Efficient Architecture Search
• Deep & Cross Network for Ad Click Predictions
• AutoFIS: Automatic Feature Interaction Selection in Factorization Models for Click-Through Rate Prediction
• Mixed Dimension Embeddings with Application to Memory-Efficient Recommendation Systems
• AutoEmb: Automated Embedding Dimensionality Search in Streaming Recommendations
• Differentiable Neural Input Search for Recommender Systems

Cited by in corpus (10)

• Click-Through Rate Prediction in Online Advertising: A Literature Review
• Embedding Compression in Recommender Systems: A Survey
• Learnable Embedding Sizes for Recommender Systems
• Adaptive Low-Precision Training for Embeddings in Click-Through Rate Prediction
• Proxy-based Item Representation for Attribute and Context-aware Recommendation
• AutoLoss: Automated Loss Function Search in Recommendations
• Pre-trained Language Model for Web-scale Retrieval in Baidu Search
• The Authors Matter: Understanding and Mitigating Implicit Bias in Deep Text Classification
• Learning Effective and Efficient Embedding via an Adaptively-Masked Twins-based Layer
• Learning Elastic Embeddings for Customizing On-Device Recommenders