Lightweight Self-Attentive Sequential Recommendation

Abstract

Modern deep neural networks (DNNs) have greatly facilitated the development of sequential recommender systems by achieving state-of-the-art recommendation performance on various sequential recommendation tasks. Given a sequence of interacted items, existing DNN-based sequential recommenders commonly embed each item into a unique vector to support subsequent computations of the user interest. However, due to the potentially large number of items, the over-parameterised item embedding matrix of a sequential recommender has become a memory bottleneck for efficient deployment in resource-constrained environments, e.g., smartphones and other edge devices. Furthermore, we observe that the widely-used multi-head self-attention, though being effective in modelling sequential dependencies among items, heavily relies on redundant attention units to fully capture both global and local item-item transition patterns within a sequence. In this paper, we introduce a novel lightweight self-attentive network (LSAN) for sequential recommendation. To aggressively compress the original embedding matrix, LSAN leverages the notion of compositional embeddings, where each item embedding is composed by merging a group of selected base embedding vectors derived from substantially smaller embedding matrices. Meanwhile, to account for the intrinsic dynamics of each item, we further propose a temporal context-aware embedding composition scheme. Besides, we develop an innovative twin-attention network that alleviates the redundancy of the traditional multi-head self-attention while retaining full capacity for capturing long- and short-term (i.e., global and local) item dependencies. Comprehensive experiments demonstrate that LSAN significantly advances the accuracy and memory efficiency of existing sequential recommenders.

References in corpus (11)

• S^3-Rec: Self-Supervised Learning for Sequential Recommendation with Mutual Information Maximization
• Predicting Dynamic Embedding Trajectory in Temporal Interaction Networks
• Translation-based Recommendation
• Rethinking the Item Order in Session-based Recommendation with Graph Neural Networks
• Pay Less Attention with Lightweight and Dynamic Convolutions
• Exploiting Cross-Session Information for Session-based Recommendation with Graph Neural Networks
• Lite Transformer with Long-Short Range Attention
• GAG: Global Attributed Graph Neural Network for Streaming Session-based Recommendation
• Exploiting Positional Information for Session-based Recommendation
• Learnable Embedding Sizes for Recommender Systems
• Learning Elastic Embeddings for Customizing On-Device Recommenders

Cited by in corpus (12)

• Contrastive Learning for Representation Degeneration Problem in Sequential Recommendation
• Intent Contrastive Learning for Sequential Recommendation
• Meta-optimized Contrastive Learning for Sequential Recommendation
• Turning Dross Into Gold Loss: is BERT4Rec really better than SASRec?
• MELT: Mutual Enhancement of Long-Tailed User and Item for Sequential Recommendation
• Exploiting Session Information in BERT-based Session-aware Sequential Recommendation
• Proxy-based Item Representation for Attribute and Context-aware Recommendation
• Collaboration and Transition: Distilling Item Transitions into Multi-Query Self-Attention for Sequential Recommendation
• Towards Lightweight Cross-domain Sequential Recommendation via External Attention-enhanced Graph Convolution Network
• Beyond Learning from Next Item: Sequential Recommendation via Personalized Interest Sustainability
• Context-based Fast Recommendation Strategy for Long User Behavior Sequence in Meituan Waimai
• Memory Augmented Multi-Instance Contrastive Predictive Coding for Sequential Recommendation