DeepLight: Deep Lightweight Feature Interactions for Accelerating CTR Predictions in Ad Serving
arXiv:2002.06987
Abstract
Click-through rate (CTR) prediction is a crucial task in online display advertising. The embedding-based neural networks have been proposed to learn both explicit feature interactions through a shallow component and deep feature interactions using a deep neural network (DNN) component. These sophisticated models, however, slow down the prediction inference by at least hundreds of times. To address the issue of significantly increased serving delay and high memory usage for ad serving in production, this paper presents \emph{DeepLight}: a framework to accelerate the CTR predictions in three aspects: 1) accelerate the model inference via explicitly searching informative feature interactions in the shallow component; 2) prune redundant layers and parameters at intra-layer and inter-layer level in the DNN component; 3) promote the sparsity of the embedding layer to preserve the most discriminant signals. By combining the above efforts, the proposed approach accelerates the model inference by 46X on Criteo dataset and 27X on Avazu dataset without any loss on the prediction accuracy. This paves the way for successfully deploying complicated embedding-based neural networks in production for ad serving.
arXiv commentAccepted by WSDM 2021; Source code: https://github.com/WayneDW/DeepLight_Deep-Lightweight-Feature-Interactions
References in corpus (26)
- Distilling the Knowledge in a Neural Network
- Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
- Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
- xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems
- Rethinking the Value of Network Pruning
- Pruning Filters for Efficient ConvNets
- DeepFM: A Factorization-Machine based Neural Network for CTR Prediction
- Speeding up Convolutional Neural Networks with Low Rank Expansions
- Learning Structured Sparsity in Deep Neural Networks
- Bayesian Compression for Deep Learning
- XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
- Submodular meets Spectral: Greedy Algorithms for Subset Selection, Sparse Approximation and Dictionary Selection
- Wide & Deep Learning for Recommender Systems
- Model compression via distillation and quantization
- Neural Factorization Machines for Sparse Predictive Analytics
- Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks
- Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy
- Good Subnetworks Provably Exist: Pruning via Greedy Forward Selection
- Product-based Neural Networks for User Response Prediction
- Higher-Order Factorization Machines
- Deep & Cross Network for Ad Click Predictions
- Deep Interest Network for Click-Through Rate Prediction
- Predicting Different Types of Conversions with Multi-Task Learning in Online Advertising
- AutoFIS: Automatic Feature Interaction Selection in Factorization Models for Click-Through Rate Prediction
- FLEN: Leveraging Field for Scalable CTR Prediction