Cross-Batch Negative Sampling for Training Two-Tower Recommenders

Abstract

The two-tower architecture has been widely applied for learning item and user representations, which is important for large-scale recommender systems. Many two-tower models are trained using various in-batch negative sampling strategies, where the effects of such strategies inherently rely on the size of mini-batches. However, training two-tower models with a large batch size is inefficient, as it demands a large volume of memory for item and user contents and consumes a lot of time for feature encoding. Interestingly, we find that neural encoders can output relatively stable features for the same input after warming up in the training process. Based on such facts, we propose a simple yet effective sampling strategy called Cross-Batch Negative Sampling (CBNS), which takes advantage of the encoded item embeddings from recent mini-batches to boost the model training. Both theoretical analysis and empirical evaluations demonstrate the effectiveness and the efficiency of CBNS.

arXiv commentPaper accepted in SIGIR'2021

References in corpus (8)

• Distributed Representations of Words and Phrases and their Compositionality
• End-to-End Neural Ad-hoc Ranking with Kernel Pooling
• Session-based Recommendations with Recurrent Neural Networks
• Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
• Graph Enhanced Representation Learning for News Recommendation
• RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering
• Multi-Interest Network with Dynamic Routing for Recommendation at Tmall
• Contrastive Learning for Debiased Candidate Generation in Large-Scale Recommender Systems