Initialization Matters: Regularizing Manifold-informed Initialization for Neural Recommendation Systems

Abstract

Proper initialization is crucial to the optimization and the generalization of neural networks. However, most existing neural recommendation systems initialize the user and item embeddings randomly. In this work, we propose a new initialization scheme for user and item embeddings called Laplacian Eigenmaps with Popularity-based Regularization for Isolated Data (LEPORID). LEPORID endows the embeddings with information regarding multi-scale neighborhood structures on the data manifold and performs adaptive regularization to compensate for high embedding variance on the tail of the data distribution. Exploiting matrix sparsity, LEPORID embeddings can be computed efficiently. We evaluate LEPORID in a wide range of neural recommendation models. In contrast to the recent surprising finding that the simple K-nearest-neighbor (KNN) method often outperforms neural recommendation systems, we show that existing neural systems initialized with LEPORID often perform on par or better than KNN. To maximize the effects of the initialization, we propose the Dual-Loss Residual Recommendation (DLR2) network, which, when initialized with LEPORID, substantially outperforms both traditional and state-of-the-art neural recommender systems.

References in corpus (12)

• LINE: Large-scale Information Network Embedding
• BPR: Bayesian Personalized Ranking from Implicit Feedback
• Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
• Disentangled Graph Collaborative Filtering
• Global Context Enhanced Graph Neural Networks for Session-based Recommendation
• Session-based Social Recommendation via Dynamic Graph Attention Networks
• Sequential Recommender Systems: Challenges, Progress and Prospects
• The Reversible Residual Network: Backpropagation Without Storing Activations
• Graph-Bert: Only Attention is Needed for Learning Graph Representations
• On the Difficulty of Evaluating Baselines: A Study on Recommender Systems
• GPT-GNN: Generative Pre-Training of Graph Neural Networks
• Bundle Recommendation with Graph Convolutional Networks