Adaptive Dense-to-Sparse Paradigm for Pruning Online Recommendation System with Non-Stationary Data
arXiv:2010.08655
Abstract
Large scale deep learning provides a tremendous opportunity to improve the quality of content recommendation systems by employing both wider and deeper models, but this comes at great infrastructural cost and carbon footprint in modern data centers. Pruning is an effective technique that reduces both memory and compute demand for model inference. However, pruning for online recommendation systems is challenging due to the continuous data distribution shift (a.k.a non-stationary data). Although incremental training on the full model is able to adapt to the non-stationary data, directly applying it on the pruned model leads to accuracy loss. This is because the sparsity pattern after pruning requires adjustment to learn new patterns. To the best of our knowledge, this is the first work to provide in-depth analysis and discussion of applying pruning to online recommendation systems with non-stationary data distribution. Overall, this work makes the following contributions: 1) We present an adaptive dense to sparse paradigm equipped with a novel pruning algorithm for pruning a large scale recommendation system with non-stationary data distribution; 2) We design the pruning algorithm to automatically learn the sparsity across layers to avoid repeating hand-tuning, which is critical for pruning the heterogeneous architectures of recommendation systems trained with non-stationary data.
References in corpus (33)
- Distilling the Knowledge in a Neural Network
- Overcoming catastrophic forgetting in neural networks
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Language Models are Few-Shot Learners
- Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- Pruning Convolutional Neural Networks for Resource Efficient Inference
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
- Rethinking the Value of Network Pruning
- PACT: Parameterized Clipping Activation for Quantized Neural Networks
- Pruning Filters for Efficient ConvNets
- A Mean Field View of the Landscape of Two-Layers Neural Networks
- A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers
- Learning Structured Sparsity in Deep Neural Networks
- Deep Learning Recommendation Model for Personalization and Recommendation Systems
- Gradient Descent Provably Optimizes Over-parameterized Neural Networks
- Model compression via distillation and quantization
- Learning the Number of Neurons in Deep Networks
- Sparse Networks from Scratch: Faster Training without Losing Performance
- Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers
- Faster CNNs with Direct Sparse Convolutions and Guided Pruning
- Global Sparse Momentum SGD for Pruning Very Deep Neural Networks
- Sequence-Level Knowledge Distillation
- Soft Threshold Weight Reparameterization for Learnable Sparsity
- Measuring the Algorithmic Efficiency of Neural Networks
- Deep Learning Inference in Facebook Data Centers: Characterization, Performance Optimizations and Hardware Implications
- What is the State of Neural Network Pruning?
- Good Subnetworks Provably Exist: Pruning via Greedy Forward Selection
- Single Shot Structured Pruning Before Training
- SparseRT: Accelerating Unstructured Sparsity on GPUs for Deep Learning Inference
- Evaluating and Characterizing Incremental Learning from Non-Stationary Data
- ShadowSync: Performing Synchronization in the Background for Highly Scalable Distributed Training