One4all User Representation for Recommender Systems in E-commerce

Abstract

General-purpose representation learning through large-scale pre-training has shown promising results in the various machine learning fields. For an e-commerce domain, the objective of general-purpose, i.e., one for all, representations would be efficient applications for extensive downstream tasks such as user profiling, targeting, and recommendation tasks. In this paper, we systematically compare the generalizability of two learning strategies, i.e., transfer learning through the proposed model, ShopperBERT, vs. learning from scratch. ShopperBERT learns nine pretext tasks with 79.2M parameters from 0.8B user behaviors collected over two years to produce user embeddings. As a result, the MLPs that employ our embedding method outperform more complex models trained from scratch for five out of six tasks. Specifically, the pre-trained embeddings have superiority over the task-specific supervised features and the strong baselines, which learn the auxiliary dataset for the cold-start problem. We also show the computational efficiency and embedding visualization of the pre-trained features.

References in corpus (10)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
• Gaussian Error Linear Units (GELUs)
• Language Models are Few-Shot Learners
• Scaling Laws for Neural Language Models
• Universal Sentence Encoder
• Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representation
• A Long-Short Demands-Aware Model for Next-Item Recommendation
• Exploiting Behavioral Consistence for Universal User Representation

Cited by in corpus (2)

• Recommender Systems in the Era of Large Language Models (LLMs)
• Scaling Law for Recommendation Models: Towards General-purpose User Representations