Contrastive Learning for Representation Degeneration Problem in Sequential Recommendation

Abstract

Recent advancements of sequential deep learning models such as Transformer and BERT have significantly facilitated the sequential recommendation. However, according to our study, the distribution of item embeddings generated by these models tends to degenerate into an anisotropic shape, which may result in high semantic similarities among embeddings. In this paper, both empirical and theoretical investigations of this representation degeneration problem are first provided, based on which a novel recommender model DuoRec is proposed to improve the item embeddings distribution. Specifically, in light of the uniformity property of contrastive learning, a contrastive regularization is designed for DuoRec to reshape the distribution of sequence representations. Given the convention that the recommendation task is performed by measuring the similarity between sequence representations and item embeddings in the same space via dot product, the regularization can be implicitly applied to the item embedding distribution. Existing contrastive learning methods mainly rely on data level augmentation for user-item interaction sequences through item cropping, masking, or reordering and can hardly provide semantically consistent augmentation samples. In DuoRec, a model-level augmentation is proposed based on Dropout to enable better semantic preserving. Furthermore, a novel sampling strategy is developed, where sequences having the same target item are chosen hard positive samples. Extensive experiments conducted on five datasets demonstrate the superior performance of the proposed DuoRec model compared with baseline methods. Visualization results of the learned representations validate that DuoRec can largely alleviate the representation degeneration problem.

References in corpus (12)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• BPR: Bayesian Personalized Ranking from Implicit Feedback
• Exploring Simple Siamese Representation Learning
• CLEAR: Contrastive Learning for Sentence Representation
• Exploiting Cross-Session Information for Session-based Recommendation with Graph Neural Networks
• COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining
• Lightweight Self-Attentive Sequential Recommendation
• Memory-augmented Dense Predictive Coding for Video Representation Learning
• Exploiting Positional Information for Session-based Recommendation
• On the Sentence Embeddings from Pre-trained Language Models
• Learning to Diversify for Single Domain Generalization
• Memory Augmented Multi-Instance Contrastive Predictive Coding for Sequential Recommendation

Cited by in corpus (40)

• TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendation
• Multi-level Cross-view Contrastive Learning for Knowledge-aware Recommender System
• Debiased Contrastive Learning for Sequential Recommendation
• Knowledge Enhancement for Contrastive Multi-Behavior Recommendation
• LinRec: Linear Attention Mechanism for Long-term Sequential Recommender Systems
• Sequential/Session-based Recommendations: Challenges, Approaches, Applications and Opportunities
• Graph Masked Autoencoder for Sequential Recommendation
• RecDCL: Dual Contrastive Learning for Recommendation
• Meta-optimized Contrastive Learning for Sequential Recommendation
• Turning Dross Into Gold Loss: is BERT4Rec really better than SASRec?
• SSLRec: A Self-Supervised Learning Framework for Recommendation
• gSASRec: Reducing Overconfidence in Sequential Recommendation Trained with Negative Sampling
• Attention Is Not the Only Choice: Counterfactual Reasoning for Path-Based Explainable Recommendation
• Multi-level Contrastive Learning Framework for Sequential Recommendation
• Spectrum-based Modality Representation Fusion Graph Convolutional Network for Multimodal Recommendation
• Dataset Regeneration for Sequential Recommendation
• Diffusion-based Contrastive Learning for Sequential Recommendation
• APGL4SR: A Generic Framework with Adaptive and Personalized Global Collaborative Information in Sequential Recommendation
• A Comprehensive Survey on Self-Supervised Learning for Recommendation
• FedDCSR: Federated Cross-domain Sequential Recommendation via Disentangled Representation Learning
• Time to Split: Exploring Data Splitting Strategies for Offline Evaluation of Sequential Recommenders
• Scalable Cross-Entropy Loss for Sequential Recommendations with Large Item Catalogs
• Group Identification via Transitional Hypergraph Convolution with Cross-view Self-supervised Learning
• Collaboration and Transition: Distilling Item Transitions into Multi-Query Self-Attention for Sequential Recommendation
• Towards Lightweight Cross-domain Sequential Recommendation via External Attention-enhanced Graph Convolution Network
• MUSE: Music Recommender System with Shuffle Play Recommendation Enhancement
• RECE: Reduced Cross-Entropy Loss for Large-Catalogue Sequential Recommenders
• CoMaPOI: A Collaborative Multi-Agent Framework for Next POI Prediction Bridging the Gap Between Trajectory and Language
• Predicting User Behavior in Smart Spaces with LLM-Enhanced Logs and Personalized Prompts
• Enhancing Transformers without Self-supervised Learning: A Loss Landscape Perspective in Sequential Recommendation
• Filtering with Time-frequency Analysis: An Adaptive and Lightweight Model for Sequential Recommender Systems Based on Discrete Wavelet Transform
• A Vlogger-augmented Graph Neural Network Model for Micro-video Recommendation
• Linear Item-Item Model with Neural Knowledge for Session-based Recommendation
• Autoregressive Generation Strategies for Top-K Sequential Recommendations
• Distribution-Guided Auto-Encoder for User Multimodal Interest Cross Fusion
• Enhancing Recommendation with Denoising Auxiliary Task
• Modality Alignment with Multi-scale Bilateral Attention for Multimodal Recommendation
• Multi-Item-Query Attention for Stable Sequential Recommendation
• SimDiffRec: Semantic Similarity-Guided Diffusion for Contrastive Sequential Recommendation
• RUEL: Retrieval-Augmented User Representation with Edge Browser Logs for Sequential Recommendation