LinRec: Linear Attention Mechanism for Long-term Sequential Recommender Systems

Abstract

Transformer models have achieved remarkable success in sequential recommender systems (SRSs). However, computing the attention matrix in traditional dot-product attention mechanisms results in a quadratic complexity with sequence lengths, leading to high computational costs for long-term sequential recommendation. Motivated by the above observation, we propose a novel L2-Normalized Linear Attention for the Transformer-based Sequential Recommender Systems (LinRec), which theoretically improves efficiency while preserving the learning capabilities of the traditional dot-product attention. Specifically, by thoroughly examining the equivalence conditions of efficient attention mechanisms, we show that LinRec possesses linear complexity while preserving the property of attention mechanisms. In addition, we reveal its latent efficiency properties by interpreting the proposed LinRec mechanism through a statistical lens. Extensive experiments are conducted based on two public benchmark datasets, demonstrating that the combination of LinRec and Transformer models achieves comparable or even superior performance than state-of-the-art Transformer-based SRS models while significantly improving time and memory efficiency.

SIGIR 2023

References in corpus (17)

• Contrastive Learning for Representation Degeneration Problem in Sequential Recommendation
• Deep Reinforcement Learning for Page-wise Recommendations
• Intent Contrastive Learning for Sequential Recommendation
• Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learning
• Towards Long-term Fairness in Recommendation
• Multi-Behavior Hypergraph-Enhanced Transformer for Sequential Recommendation
• Augmenting Sequential Recommendation with Pseudo-Prior Items via Reversely Pre-training Transformer
• Neural Interactive Collaborative Filtering
• CauseRec: Counterfactual User Sequence Synthesis for Sequential Recommendation
• Jointly Learning to Recommend and Advertise
• AutoMLP: Automated MLP for Sequential Recommendations
• Long-Short Transformer: Efficient Transformers for Language and Vision
• Multi-Task Recommendations with Reinforcement Learning
• Variational Reasoning over Incomplete Knowledge Graphs for Conversational Recommendation
• Exploration and Regularization of the Latent Action Space in Recommendation
• Generating Counterfactual Hard Negative Samples for Graph Contrastive Learning
• User Retention-oriented Recommendation with Decision Transformer

Cited by in corpus (6)

• Sequential Recommendation for Optimizing Both Immediate Feedback and Long-term Retention
• Efficient and Robust Regularized Federated Recommendation
• AutoAssign+: Automatic Shared Embedding Assignment in Streaming Recommendation
• Multi-task Offline Reinforcement Learning for Online Advertising in Recommender Systems
• Modeling Long-term User Behaviors with Diffusion-driven Multi-interest Network for CTR Prediction
• Zeroshot Listwise Learning to Rank Algorithm for Recommendation