Industry Insights from Comparing Deep Learning and GBDT Models for E-Commerce Learning-to-Rank

Abstract

In e-commerce recommender and search systems, tree-based models, such as LambdaMART, have set a strong baseline for Learning-to-Rank (LTR) tasks. Despite their effectiveness and widespread adoption in industry, the debate continues whether deep neural networks (DNNs) can outperform traditional tree-based models in this domain. To contribute to this discussion, we systematically benchmark DNNs against our production-grade LambdaMART model. We evaluate multiple DNN architectures and loss functions on a proprietary dataset from OTTO and validate our findings through an 8-week online A/B test. The results show that a simple DNN architecture outperforms a strong tree-based baseline in terms of total clicks and revenue, while achieving parity in total units sold.

This work was accepted for publication in the 19th ACM Conference on Recommender Systems (RecSys 2025). The final published version will be available at the ACM Digital Library

References in corpus (7)

• On Application of Learning to Rank for E-Commerce Search
• Applying Deep Learning To Airbnb Search
• When Inverse Propensity Scoring does not Work: Affine Corrections for Unbiased Learning to Rank
• Widespread Flaws in Offline Evaluation of Recommender Systems
• Scaling Session-Based Transformer Recommendations using Optimized Negative Sampling and Loss Functions
• RankFormer: Listwise Learning-to-Rank Using Listwide Labels
• Unbiased Learning to Rank Meets Reality: Lessons from Baidu's Large-Scale Search Dataset