Ranking Distillation: Learning Compact Ranking Models With High Performance for Recommender System

Abstract

We propose a novel way to train ranking models, such as recommender systems, that are both effective and efficient. Knowledge distillation (KD) was shown to be successful in image recognition to achieve both effectiveness and efficiency. We propose a KD technique for learning to rank problems, called \emph{ranking distillation (RD)}. Specifically, we train a smaller student model to learn to rank documents/items from both the training data and the supervision of a larger teacher model. The student model achieves a similar ranking performance to that of the large teacher model, but its smaller model size makes the online inference more efficient. RD is flexible because it is orthogonal to the choices of ranking models for the teacher and student. We address the challenges of RD for ranking problems. The experiments on public data sets and state-of-the-art recommendation models showed that RD achieves its design purposes: the student model learnt with RD has a model size less than half of the teacher model while achieving a ranking performance similar to the teacher model and much better than the student model learnt without RD.

arXiv commentAccepted at KDD 2018

References in corpus (13)

• Distilling the Knowledge in a Neural Network
• BPR: Bayesian Personalized Ranking from Implicit Feedback
• FitNets: Hints for Thin Deep Nets
• Do Deep Nets Really Need to be Deep?
• Convolutional Neural Networks for Sentence Classification
• The Loss Surfaces of Multilayer Networks
• End-to-End Neural Ad-hoc Ranking with Kernel Pooling
• Neural Collaborative Filtering
• Text Matching as Image Recognition
• Large scale distributed neural network training through online distillation
• Sequence-Level Knowledge Distillation
• Fusing Similarity Models with Markov Chains for Sparse Sequential Recommendation
• Neural Ranking Models with Weak Supervision

Cited by in corpus (7)

• Knowledge Distillation: A Survey
• Deep Learning based Recommender System: A Survey and New Perspectives
• Graph-less Collaborative Filtering
• Collaboration and Transition: Distilling Item Transitions into Multi-Query Self-Attention for Sequential Recommendation
• AIBench Scenario: Scenario-distilling AI Benchmarking
• Dual Correction Strategy for Ranking Distillation in Top-N Recommender System
• Topology Distillation for Recommender System