Pretrained Embeddings for E-commerce Machine Learning: When it Fails and Why?

Abstract

The use of pretrained embeddings has become widespread in modern e-commerce machine learning (ML) systems. In practice, however, we have encountered several key issues when using pretrained embedding in a real-world production system, many of which cannot be fully explained by current knowledge. Unfortunately, we find that there is a lack of a thorough understanding of how pre-trained embeddings work, especially their intrinsic properties and interactions with downstream tasks. Consequently, it becomes challenging to make interactive and scalable decisions regarding the use of pre-trained embeddings in practice. Our investigation leads to two significant discoveries about using pretrained embeddings in e-commerce applications. Firstly, we find that the design of the pretraining and downstream models, particularly how they encode and decode information via embedding vectors, can have a profound impact. Secondly, we establish a principled perspective of pre-trained embeddings via the lens of kernel analysis, which can be used to evaluate their predictability, interactively and scalably. These findings help to address the practical challenges we faced and offer valuable guidance for successful adoption of pretrained embeddings in real-world production. Our conclusions are backed by solid theoretical reasoning, benchmark experiments, as well as online testings.

References in corpus (12)

• Distributed Representations of Sentences and Documents
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Facial Expression Analysis under Partial Occlusion: A Survey
• Domain Generalization: A Survey
• Towards Out-Of-Distribution Generalization: A Survey
• Meta-Prod2Vec - Product Embeddings Using Side-Information for Recommendation
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Compositional generalization in a deep seq2seq model by separating syntax and semantics
• Deep Learning for Sequential Recommendation: Algorithms, Influential Factors, and Evaluations
• Product Knowledge Graph Embedding for E-commerce
• AliMe KG: Domain Knowledge Graph Construction and Application in E-commerce
• Knowledge-aware Complementary Product Representation Learning

Cited by in corpus (1)

• EMS: Efficient and Effective Massively Multilingual Sentence Embedding Learning