Churn Reduction via Distillation

Abstract

In real-world systems, models are frequently updated as more data becomes available, and in addition to achieving high accuracy, the goal is to also maintain a low difference in predictions compared to the base model (i.e. predictive "churn"). If model retraining results in vastly different behavior, then it could cause negative effects in downstream systems, especially if this churn can be avoided with limited impact on model accuracy. In this paper, we show an equivalence between training with distillation using the base model as the teacher and training with an explicit constraint on the predictive churn. We then show that distillation performs strongly for low churn training against a number of recent baselines on a wide range of datasets and model architectures, including fully-connected networks, convolutional networks, and transformers.

References in corpus (18)

• Distilling the Knowledge in a Neural Network
• mixup: Beyond Empirical Risk Minimization
• Model compression via distillation and quantization
• How Does Batch Normalization Help Optimization?
• Harnessing Deep Neural Networks with Logic Rules
• Online Batch Selection for Faster Training of Neural Networks
• Unifying distillation and privileged information
• Large scale distributed neural network training through online distillation
• Towards Understanding Knowledge Distillation
• Satisfying Real-world Goals with Dataset Constraints
• Self-Distillation Amplifies Regularization in Hilbert Space
• On Warm-Starting Neural Network Training
• Surrogate Regret Bounds for Bipartite Ranking via Strongly Proper Losses
• Distillation $\approx$ Early Stopping? Harvesting Dark Knowledge Utilizing Anisotropic Information Retrieval For Overparameterized Neural Network
• On the Reproducibility of Neural Network Predictions
• Why distillation helps: a statistical perspective
• Hypothesis Set Stability and Generalization
• Locally Adaptive Label Smoothing for Predictive Churn