Collaboration and Transition: Distilling Item Transitions into Multi-Query Self-Attention for Sequential Recommendation

Abstract

Modern recommender systems employ various sequential modules such as self-attention to learn dynamic user interests. However, these methods are less effective in capturing collaborative and transitional signals within user interaction sequences. First, the self-attention architecture uses the embedding of a single item as the attention query, making it challenging to capture collaborative signals. Second, these methods typically follow an auto-regressive framework, which is unable to learn global item transition patterns. To overcome these limitations, we propose a new method called Multi-Query Self-Attention with Transition-Aware Embedding Distillation (MQSA-TED). First, we propose an $L$-query self-attention module that employs flexible window sizes for attention queries to capture collaborative signals. In addition, we introduce a multi-query self-attention method that balances the bias-variance trade-off in modeling user preferences by combining long and short-query self-attentions. Second, we develop a transition-aware embedding distillation module that distills global item-to-item transition patterns into item embeddings, which enables the model to memorize and leverage transitional signals and serves as a calibrator for collaborative signals. Experimental results on four real-world datasets demonstrate the effectiveness of the proposed modules.

WSDM 2024 Oral Presentation

References in corpus (14)

• Adam: A Method for Stochastic Optimization
• Distilling the Knowledge in a Neural Network
• Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering
• S^3-Rec: Self-Supervised Learning for Sequential Recommendation with Mutual Information Maximization
• Contrastive Learning for Representation Degeneration Problem in Sequential Recommendation
• Translation-based Recommendation
• LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation
• Filter-enhanced MLP is All You Need for Sequential Recommendation
• BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer
• Towards Representation Alignment and Uniformity in Collaborative Filtering
• KuaiRand: An Unbiased Sequential Recommendation Dataset with Randomly Exposed Videos
• Lightweight Self-Attentive Sequential Recommendation
• Sequential Recommendation with Self-Attentive Multi-Adversarial Network
• Ranking Distillation: Learning Compact Ranking Models With High Performance for Recommender System