ENTMOOT: A Framework for Optimization over Ensemble Tree Models

Abstract

Gradient boosted trees and other regression tree models perform well in a wide range of real-world, industrial applications. These tree models (i) offer insight into important prediction features, (ii) effectively manage sparse data, and (iii) have excellent prediction capabilities. Despite their advantages, they are generally unpopular for decision-making tasks and black-box optimization, which is due to their difficult-to optimize structure and the lack of a reliable uncertainty measure. ENTMOOT is our new framework for integrating (already trained) tree models into larger optimization problems. The contributions of ENTMOOT include: (i) explicitly introducing a reliable uncertainty measure that is compatible with tree models, (ii) solving the larger optimization problems that incorporate these uncertainty aware tree models, (iii) proving that the solutions are globally optimal, i.e. no better solution exists. In particular, we show how the ENTMOOT approach allows a simple integration of tree models into decision-making and black-box optimization, where it proves as a strong competitor to commonly-used frameworks.

arXiv comment33 pages, 10 figures, 2 tables

References in corpus (15)

• Scikit-learn: Machine Learning in Python
• XGBoost: A Scalable Tree Boosting System
• Practical Bayesian Optimization of Machine Learning Algorithms
• A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
• Scalable Bayesian Optimization Using Deep Neural Networks
• A Tutorial on Bayesian Optimization
• Bayesian optimization for materials design
• The ALAMO approach to machine learning
• Global Deterministic Optimization with Artificial Neural Networks Embedded
• Scalable Global Optimization via Local Bayesian Optimization
• High Dimensional Bayesian Optimisation and Bandits via Additive Models
• Task-based End-to-end Model Learning in Stochastic Optimization
• Batched Large-scale Bayesian Optimization in High-dimensional Spaces
• BOCK : Bayesian Optimization with Cylindrical Kernels
• Active Learning of Linear Embeddings for Gaussian Processes

Cited by in corpus (2)

• Maximizing information from chemical engineering data sets: Applications to machine learning
• Multi-Objective Constrained Optimization for Energy Applications via Tree Ensembles