Unbiased Knowledge Distillation for Recommendation

Abstract

As a promising solution for model compression, knowledge distillation (KD) has been applied in recommender systems (RS) to reduce inference latency. Traditional solutions first train a full teacher model from the training data, and then transfer its knowledge (\ie \textit{soft labels}) to supervise the learning of a compact student model. However, we find such a standard distillation paradigm would incur serious bias issue -- popular items are more heavily recommended after the distillation. This effect prevents the student model from making accurate and fair recommendations, decreasing the effectiveness of RS. In this work, we identify the origin of the bias in KD -- it roots in the biased soft labels from the teacher, and is further propagated and intensified during the distillation. To rectify this, we propose a new KD method with a stratified distillation strategy. It first partitions items into multiple groups according to their popularity, and then extracts the ranking knowledge within each group to supervise the learning of the student. Our method is simple and teacher-agnostic -- it works on distillation stage without affecting the training of the teacher model. We conduct extensive theoretical and empirical studies to validate the effectiveness of our proposal. We release our code at: https://github.com/chengang95/UnKD.

References in corpus (13)

• BPR: Bayesian Personalized Ranking from Implicit Feedback
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Causal Intervention for Leveraging Popularity Bias in Recommendation
• LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation
• Model-Agnostic Counterfactual Reasoning for Eliminating Popularity Bias in Recommender System
• Causal Embeddings for Recommendation
• Collaborative Filtering and the Missing at Random Assumption
• AutoDebias: Learning to Debias for Recommendation
• Bias and Debias in Recommender System: A Survey and Future Directions
• DE-RRD: A Knowledge Distillation Framework for Recommender System
• Should Graph Convolution Trust Neighbors? A Simple Causal Inference Method
• Bidirectional Distillation for Top-K Recommender System
• Crank up the volume: preference bias amplification in collaborative recommendation

Cited by in corpus (4)

• Distillation Matters: Empowering Sequential Recommenders to Match the Performance of Large Language Model
• LightGNN: Simple Graph Neural Network for Recommendation
• Dynamic Sparse Learning: A Novel Paradigm for Efficient Recommendation
• Linear Item-Item Model with Neural Knowledge for Session-based Recommendation