Scaling Law for Recommendation Models: Towards General-purpose User Representations

Abstract

Recent advancement of large-scale pretrained models such as BERT, GPT-3, CLIP, and Gopher, has shown astonishing achievements across various task domains. Unlike vision recognition and language models, studies on general-purpose user representation at scale still remain underexplored. Here we explore the possibility of general-purpose user representation learning by training a universal user encoder at large scales. We demonstrate that the scaling law is present in user representation learning areas, where the training error scales as a power-law with the amount of computation. Our Contrastive Learning User Encoder (CLUE), optimizes task-agnostic objectives, and the resulting user embeddings stretch our expectation of what is possible to do in various downstream tasks. CLUE also shows great transferability to other domains and companies, as performances on an online experiment shows significant improvements in Click-Through-Rate (CTR). Furthermore, we also investigate how the model performance is influenced by the scale factors, such as training data size, model capacity, sequence length, and batch size. Finally, we discuss the broader impacts of CLUE in general.

arXiv commentAccepted at AAAI 2023. This version includes the technical appendix

References in corpus (21)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Decoupled Weight Decay Regularization
• Learning Transferable Visual Models From Natural Language Supervision
• On the difficulty of training Recurrent Neural Networks
• Language Models are Few-Shot Learners
• On the Opportunities and Risks of Foundation Models
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Scaling Laws for Neural Language Models
• Zero-Shot Text-to-Image Generation
• Mixed Precision Training
• DeepFM: A Factorization-Machine based Neural Network for CTR Prediction
• Explaining Neural Scaling Laws
• NSML: Meet the MLaaS platform with a real-world case study
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• NSML: A Machine Learning Platform That Enables You to Focus on Your Models
• Representation Learning via Invariant Causal Mechanisms
• Behavior Sequence Transformer for E-commerce Recommendation in Alibaba
• One4all User Representation for Recommender Systems in E-commerce
• Learning Curve Theory
• Interest-oriented Universal User Representation via Contrastive Learning
• div2vec: Diversity-Emphasized Node Embedding