Initialization Matters: Regularizing Manifold-informed Initialization for Neural Recommendation Systems

Abstract

Proper initialization is crucial to the optimization and the generalization of neural networks. However, most existing neural recommendation systems initialize the user and item embeddings randomly. In this work, we propose a new initialization scheme for user and item embeddings called Laplacian Eigenmaps with Popularity-based Regularization for Isolated Data (LEPORID). LEPORID endows the embeddings with information regarding multi-scale neighborhood structures on the data manifold and performs adaptive regularization to compensate for high embedding variance on the tail of the data distribution. Exploiting matrix sparsity, LEPORID embeddings can be computed efficiently. We evaluate LEPORID in a wide range of neural recommendation models. In contrast to the recent surprising finding that the simple K-nearest-neighbor (KNN) method often outperforms neural recommendation systems, we show that existing neural systems initialized with LEPORID often perform on par or better than KNN. To maximize the effects of the initialization, we propose the Dual-Loss Residual Recommendation (DLR2) network, which, when initialized with LEPORID, substantially outperforms both traditional and state-of-the-art neural recommender systems.

References in corpus (26)

• Adam: A Method for Stochastic Optimization
• Statistical mechanics of complex networks
• DeepWalk: Online Learning of Social Representations
• LINE: Large-scale Information Network Embedding
• Deep Residual Learning for Image Recognition
• BPR: Bayesian Personalized Ranking from Implicit Feedback
• Neural Graph Collaborative Filtering
• Graph Convolutional Neural Networks for Web-Scale Recommender Systems
• Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
• PTE: Predictive Text Embedding through Large-scale Heterogeneous Text Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Disentangled Graph Collaborative Filtering
• Global Context Enhanced Graph Neural Networks for Session-based Recommendation
• Session-based Recommendations with Recurrent Neural Networks
• Session-based Social Recommendation via Dynamic Graph Attention Networks
• Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches
• Sequential Recommender Systems: Challenges, Progress and Prospects
• The Reversible Residual Network: Backpropagation Without Storing Activations
• Invertible Residual Networks
• Graph-Bert: Only Attention is Needed for Learning Graph Representations
• Graph Neural Networks in Recommender Systems: A Survey
• On the Difficulty of Evaluating Baselines: A Study on Recommender Systems
• Self-Attentive Sequential Recommendation
• GPT-GNN: Generative Pre-Training of Graph Neural Networks
• Bundle Recommendation with Graph Convolutional Networks
• A Fast Hierarchically Preconditioned Eigensolver Based On Multiresolution Matrix Decomposition