Dual Correction Strategy for Ranking Distillation in Top-N Recommender System

Abstract

Knowledge Distillation (KD), which transfers the knowledge of a well-trained large model (teacher) to a small model (student), has become an important area of research for practical deployment of recommender systems. Recently, Relaxed Ranking Distillation (RRD) has shown that distilling the ranking information in the recommendation list significantly improves the performance. However, the method still has limitations in that 1) it does not fully utilize the prediction errors of the student model, which makes the training not fully efficient, and 2) it only distills the user-side ranking information, which provides an insufficient view under the sparse implicit feedback. This paper presents Dual Correction strategy for Distillation (DCD), which transfers the ranking information from the teacher model to the student model in a more efficient manner. Most importantly, DCD uses the discrepancy between the teacher model and the student model predictions to decide which knowledge to be distilled. By doing so, DCD essentially provides the learning guidance tailored to "correcting" what the student model has failed to accurately predict. This process is applied for transferring the ranking information from the user-side as well as the item-side to address sparse implicit user feedback. Our experiments show that the proposed method outperforms the state-of-the-art baselines, and ablation studies validate the effectiveness of each component.

CIKM 2021

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Distilling the Knowledge in a Neural Network
• BPR: Bayesian Personalized Ranking from Implicit Feedback
• FitNets: Hints for Thin Deep Nets
• Born Again Neural Networks
• DE-RRD: A Knowledge Distillation Framework for Recommender System
• Bidirectional Distillation for Top-K Recommender System
• Binarized Collaborative Filtering with Distilling Graph Convolutional Networks
• Unsupervised Proxy Selection for Session-based Recommender Systems
• Ranking Distillation: Learning Compact Ranking Models With High Performance for Recommender System
• Discrete Factorization Machines for Fast Feature-based Recommendation