Lightweight Self-Attentive Sequential Recommendation

Abstract

Modern deep neural networks (DNNs) have greatly facilitated the development of sequential recommender systems by achieving state-of-the-art recommendation performance on various sequential recommendation tasks. Given a sequence of interacted items, existing DNN-based sequential recommenders commonly embed each item into a unique vector to support subsequent computations of the user interest. However, due to the potentially large number of items, the over-parameterised item embedding matrix of a sequential recommender has become a memory bottleneck for efficient deployment in resource-constrained environments, e.g., smartphones and other edge devices. Furthermore, we observe that the widely-used multi-head self-attention, though being effective in modelling sequential dependencies among items, heavily relies on redundant attention units to fully capture both global and local item-item transition patterns within a sequence. In this paper, we introduce a novel lightweight self-attentive network (LSAN) for sequential recommendation. To aggressively compress the original embedding matrix, LSAN leverages the notion of compositional embeddings, where each item embedding is composed by merging a group of selected base embedding vectors derived from substantially smaller embedding matrices. Meanwhile, to account for the intrinsic dynamics of each item, we further propose a temporal context-aware embedding composition scheme. Besides, we develop an innovative twin-attention network that alleviates the redundancy of the traditional multi-head self-attention while retaining full capacity for capturing long- and short-term (i.e., global and local) item dependencies. Comprehensive experiments demonstrate that LSAN significantly advances the accuracy and memory efficiency of existing sequential recommenders.

References in corpus (8)

• S^3-Rec: Self-Supervised Learning for Sequential Recommendation with Mutual Information Maximization
• Translation-based Recommendation
• Pay Less Attention with Lightweight and Dynamic Convolutions
• Exploiting Cross-Session Information for Session-based Recommendation with Graph Neural Networks
• Lite Transformer with Long-Short Range Attention
• Exploiting Positional Information for Session-based Recommendation
• Learnable Embedding Sizes for Recommender Systems
• Learning Elastic Embeddings for Customizing On-Device Recommenders

Cited by in corpus (3)

• Intent Contrastive Learning for Sequential Recommendation
• Exploiting Session Information in BERT-based Session-aware Sequential Recommendation
• Memory Augmented Multi-Instance Contrastive Predictive Coding for Sequential Recommendation