Differentiable Neural Input Search for Recommender Systems

Abstract

Latent factor models are the driving forces of the state-of-the-art recommender systems, with an important insight of vectorizing raw input features into dense embeddings. The dimensions of different feature embeddings are often set to a same value empirically, which limits the predictive performance of latent factor models. Existing works have proposed heuristic or reinforcement learning-based methods to search for mixed feature embedding dimensions. For efficiency concern, these methods typically choose embedding dimensions from a restricted set of candidate dimensions. However, this restriction will hurt the flexibility of dimension selection, leading to suboptimal performance of search results. In this paper, we propose Differentiable Neural Input Search (DNIS), a method that searches for mixed feature embedding dimensions in a more flexible space through continuous relaxation and differentiable optimization. The key idea is to introduce a soft selection layer that controls the significance of each embedding dimension, and optimize this layer according to model's validation performance. DNIS is model-agnostic and thus can be seamlessly incorporated with existing latent factor models for recommendation. We conduct experiments with various architectures of latent factor models on three public real-world datasets for rating prediction, Click-Through-Rate (CTR) prediction, and top-k item recommendation. The results demonstrate that our method achieves the best predictive performance compared with existing neural input search approaches with fewer embedding parameters and less time cost.

References in corpus (12)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• DARTS: Differentiable Architecture Search
• xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Large-Scale Evolution of Image Classifiers
• Designing Neural Network Architectures using Reinforcement Learning
• Gradient-based Hyperparameter Optimization through Reversible Learning
• Searching for Efficient Multi-Scale Architectures for Dense Image Prediction
• Hyperparameter optimization with approximate gradient
• Deep Learning Inference in Facebook Data Centers: Characterization, Performance Optimizations and Hardware Implications
• Mixed Dimension Embeddings with Application to Memory-Efficient Recommendation Systems
• AutoEmb: Automated Embedding Dimensionality Search in Streaming Recommendations

Cited by in corpus (11)

• AutoML: A Survey of the State-of-the-Art
• Click-Through Rate Prediction in Online Advertising: A Literature Review
• OptEmbed: Learning Optimal Embedding Table for Click-through Rate Prediction
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Embedding Compression in Recommender Systems: A Survey
• Learnable Embedding Sizes for Recommender Systems
• Memory-efficient Embedding for Recommendations
• Dynamic Embedding Size Search with Minimum Regret for Streaming Recommender System
• i-Razor: A Differentiable Neural Input Razor for Feature Selection and Dimension Search in DNN-Based Recommender Systems
• AutoLoss: Automated Loss Function Search in Recommendations
• Random Offset Block Embedding Array (ROBE) for CriteoTB Benchmark MLPerf DLRM Model : 1000$\times$ Compression and 3.1$\times$ Faster Inference