CatBoost: unbiased boosting with categorical features

Abstract

This paper presents the key algorithmic techniques behind CatBoost, a new gradient boosting toolkit. Their combination leads to CatBoost outperforming other publicly available boosting implementations in terms of quality on a variety of datasets. Two critical algorithmic advances introduced in CatBoost are the implementation of ordered boosting, a permutation-driven alternative to the classic algorithm, and an innovative algorithm for processing categorical features. Both techniques were created to fight a prediction shift caused by a special kind of target leakage present in all currently existing implementations of gradient boosting algorithms. In this paper, we provide a detailed analysis of this problem and demonstrate that proposed algorithms solve it effectively, leading to excellent empirical results.

References in corpus (2)

• Boosted Decision Trees as an Alternative to Artificial Neural Networks for Particle Identification
• Enhancing LambdaMART Using Oblivious Trees

Cited by in corpus (23)

• A Prescriptive Learning Analytics Framework: Beyond Predictive Modelling and onto Explainable AI with Prescriptive Analytics and ChatGPT
• Artificial Intelligence-Based Analytics for Impacts of COVID-19 and Online Learning on College Students' Mental Health
• Tabular Machine Learning Methods for Predicting Gas Turbine Emissions
• Enhancing operations management through smart sensors: measuring and improving well-being, interaction and performance of logistics workers
• Development and Comparative Analysis of Machine Learning Models for Hypoxemia Severity Triage in CBRNE Emergency Scenarios Using Physiological and Demographic Data from Medical-Grade Devices
• Boosting Algorithms for Delivery Time Prediction in Transportation Logistics
• PaloBoost: An Overfitting-robust TreeBoost with Out-of-Bag Sample Regularization Techniques
• A High-Throughput and Data-Driven Computational Framework for Novel Quantum Materials
• Trees-Based Models for Correlated Data
• TreeFlow: Going beyond Tree-based Gaussian Probabilistic Regression
• Predicting Foreign Exchange EUR/USD direction using machine learning
• Tab-Attention: Self-Attention-based Stacked Generalization for Imbalanced Credit Default Prediction
• Is it feasible to detect FLOSS version release events from textual messages? A case study on Stack Overflow
• CryptoAnalytics: Cryptocoins Price Forecasting with Machine Learning Techniques
• CatBoost model with synthetic features in application to loan risk assessment of small businesses
• Contrastive Mixup: Self- and Semi-Supervised learning for Tabular Domain
• MementoML: Performance of selected machine learning algorithm configurations on OpenML100 datasets
• Inductive inference of gradient-boosted decision trees on graphs for insurance fraud detection
• Solving Fashion Recommendation -- The Farfetch Challenge
• WildWood: a new Random Forest algorithm
• Natural Attribute-based Shift Detection
• Microcircuit synchronization and heavy tailed synaptic weight distribution in preBötzinger Complex contribute to generation of breathing rhythm
• Cost-effective vibration analysis through data-backed pipeline optimisation