BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer

Abstract

Modeling users' dynamic and evolving preferences from their historical behaviors is challenging and crucial for recommendation systems. Previous methods employ sequential neural networks (e.g., Recurrent Neural Network) to encode users' historical interactions from left to right into hidden representations for making recommendations. Although these methods achieve satisfactory results, they often assume a rigidly ordered sequence which is not always practical. We argue that such left-to-right unidirectional architectures restrict the power of the historical sequence representations. For this purpose, we introduce a Bidirectional Encoder Representations from Transformers for sequential Recommendation (BERT4Rec). However, jointly conditioning on both left and right context in deep bidirectional model would make the training become trivial since each item can indirectly "see the target item". To address this problem, we train the bidirectional model using the Cloze task, predicting the masked items in the sequence by jointly conditioning on their left and right context. Comparing with predicting the next item at each position in a sequence, the Cloze task can produce more samples to train a more powerful bidirectional model. Extensive experiments on four benchmark datasets show that our model outperforms various state-of-the-art sequential models consistently.

To appear in CIKM 2019

References in corpus (8)

• Efficient Estimation of Word Representations in Vector Space
• Distributed Representations of Words and Phrases and their Compositionality
• Distilling the Knowledge in a Neural Network
• BPR: Bayesian Personalized Ranking from Implicit Feedback
• A Structured Self-attentive Sentence Embedding
• Recurrent Neural Networks with Top-k Gains for Session-based Recommendations
• Layer Normalization
• Translation-based Recommendation

Cited by in corpus (86)

• A General Survey on Attention Mechanisms in Deep Learning
• Filter-enhanced MLP is All You Need for Sequential Recommendation
• Fast Parallel Hypertree Decompositions in Logarithmic Recursion Depth
• Uncovering ChatGPT's Capabilities in Recommender Systems
• Knowledge Graph Self-Supervised Rationalization for Recommendation
• Debiased Contrastive Learning for Sequential Recommendation
• Multi-Behavior Hypergraph-Enhanced Transformer for Sequential Recommendation
• Large Language Models as Zero-Shot Conversational Recommenders
• Leveraging Large Language Models for Sequential Recommendation
• Disentangling Long and Short-Term Interests for Recommendation
• Social Recommendation with Self-Supervised Metagraph Informax Network
• Contrastive Cross-Domain Sequential Recommendation
• CARCA: Context and Attribute-Aware Next-Item Recommendation via Cross-Attention
• Multi-Behavior Sequential Recommendation with Temporal Graph Transformer
• Price DOES Matter! Modeling Price and Interest Preferences in Session-based Recommendation
• Multi-Behavior Graph Neural Networks for Recommender System
• Time-aware Path Reasoning on Knowledge Graph for Recommendation
• Graph Masked Autoencoder for Sequential Recommendation
• MEANTIME: Mixture of Attention Mechanisms with Multi-temporal Embeddings for Sequential Recommendation
• AutoMLP: Automated MLP for Sequential Recommendations
• Turning Dross Into Gold Loss: is BERT4Rec really better than SASRec?
• Contrastive Learning for Recommender System
• Multi-level Contrastive Learning Framework for Sequential Recommendation
• Exploration and Regularization of the Latent Action Space in Recommendation
• Deep Interest Highlight Network for Click-Through Rate Prediction in Trigger-Induced Recommendation
• Exploring Adapter-based Transfer Learning for Recommender Systems: Empirical Studies and Practical Insights
• Exploiting Positional Information for Session-based Recommendation
• Behavior Sequence Transformer for E-commerce Recommendation in Alibaba
• It Is Different When Items Are Older: Debiasing Recommendations When Selection Bias and User Preferences Are Dynamic
• User Retention-oriented Recommendation with Decision Transformer
• Contrastive Multi-Level Graph Neural Networks for Session-based Recommendation
• Modeling Users' Contextualized Page-wise Feedback for Click-Through Rate Prediction in E-commerce Search
• Debiasing Learning for Membership Inference Attacks Against Recommender Systems
• Widespread Flaws in Offline Evaluation of Recommender Systems
• StackRec: Efficient Training of Very Deep Sequential Recommender Models by Iterative Stacking
• Scenario-Adaptive and Self-Supervised Model for Multi-Scenario Personalized Recommendation
• Understanding and Modeling Passive-Negative Feedback for Short-video Sequential Recommendation
• Counterfactual Collaborative Reasoning
• The Effect of Third Party Implementations on Reproducibility
• Revisiting Neural Retrieval on Accelerators
• Exploiting Session Information in BERT-based Session-aware Sequential Recommendation
• Neighborhood-based Hard Negative Mining for Sequential Recommendation
• Meta-Learning with Adaptive Weighted Loss for Imbalanced Cold-Start Recommendation
• On the Effectiveness of Unlearning in Session-Based Recommendation
• Debiasing Sequential Recommenders through Distributionally Robust Optimization over System Exposure
• A Scalable Framework for Automatic Playlist Continuation on Music Streaming Services
• Dual-interest Factorization-heads Attention for Sequential Recommendation
• RecJPQ: Training Large-Catalogue Sequential Recommenders
• DraftRec: Personalized Draft Recommendation for Winning in Multi-Player Online Battle Arena Games
• FATA-Trans: Field And Time-Aware Transformer for Sequential Tabular Data
• Proxy-based Item Representation for Attribute and Context-aware Recommendation
• Collaboration and Transition: Distilling Item Transitions into Multi-Query Self-Attention for Sequential Recommendation
• Debiasing the Cloze Task in Sequential Recommendation with Bidirectional Transformers
• Capturing Popularity Trends: A Simplistic Non-Personalized Approach for Enhanced Item Recommendation
• Entire Space Multi-Task Modeling via Post-Click Behavior Decomposition for Conversion Rate Prediction
• A Generic Network Compression Framework for Sequential Recommender Systems
• CPMR: Context-Aware Incremental Sequential Recommendation with Pseudo-Multi-Task Learning
• Query-dominant User Interest Network for Large-Scale Search Ranking
• AMEIR: Automatic Behavior Modeling, Interaction Exploration and MLP Investigation in the Recommender System
• Predicting User Behavior in Smart Spaces with LLM-Enhanced Logs and Personalized Prompts
• Superbloom: Bloom filter meets Transformer
• Inter-sequence Enhanced Framework for Personalized Sequential Recommendation
• Accurate Bundle Matching and Generation via Multitask Learning with Partially Shared Parameters
• Improving End-to-End Sequential Recommendations with Intent-aware Diversification
• Towards Validating Long-Term User Feedbacks in Interactive Recommendation Systems
• Enhancing Transformers without Self-supervised Learning: A Loss Landscape Perspective in Sequential Recommendation
• Learn Basic Skills and Reuse: Modularized Adaptive Neural Architecture Search (MANAS)
• Learning to Structure Long-term Dependence for Sequential Recommendation
• It's Enough: Relaxing Diagonal Constraints in Linear Autoencoders for Recommendation
• Feature Staleness Aware Incremental Learning for CTR Prediction
• CSSR: A Context-Aware Sequential Software Service Recommendation Model
• Deep Unified Representation for Heterogeneous Recommendation
• Improving Next-Application Prediction with Deep Personalized-Attention Neural Network
• Match4Rec: A Novel Recommendation Algorithm Based on Bidirectional Encoder Representation with the Matching Task
• Autoregressive Generation Strategies for Top-K Sequential Recommendations
• Label Attention Network for Temporal Sets Prediction: You Were Looking at a Wrong Self-Attention
• POG: Personalized Outfit Generation for Fashion Recommendation at Alibaba iFashion
• A News Recommender System Considering Temporal Dynamics and Diversity
• Intrinsic and Extrinsic Factor Disentanglement for Recommendation in Various Context Scenarios
• Reusable Self-Attention Recommender Systems in Fashion Industry Applications
• Data Augmentation Using Many-To-Many RNNs for Session-Aware Recommender Systems
• RUEL: Retrieval-Augmented User Representation with Edge Browser Logs for Sequential Recommendation
• InfoBehavior: Self-supervised Representation Learning for Ultra-long Behavior Sequence via Hierarchical Grouping
• Breaking the Softmax Bottleneck for Sequential Recommender Systems with Dropout and Decoupling
• Preliminary Evaluation of the Test-Time Training Layers in Recommendation System (Student Abstract)
• Sequential Recommendation for Cold-start Users with Meta Transitional Learning