ENTMOOT: A Framework for Optimization over Ensemble Tree Models

Abstract

Gradient boosted trees and other regression tree models perform well in a wide range of real-world, industrial applications. These tree models (i) offer insight into important prediction features, (ii) effectively manage sparse data, and (iii) have excellent prediction capabilities. Despite their advantages, they are generally unpopular for decision-making tasks and black-box optimization, which is due to their difficult-to optimize structure and the lack of a reliable uncertainty measure. ENTMOOT is our new framework for integrating (already trained) tree models into larger optimization problems. The contributions of ENTMOOT include: (i) explicitly introducing a reliable uncertainty measure that is compatible with tree models, (ii) solving the larger optimization problems that incorporate these uncertainty aware tree models, (iii) proving that the solutions are globally optimal, i.e. no better solution exists. In particular, we show how the ENTMOOT approach allows a simple integration of tree models into decision-making and black-box optimization, where it proves as a strong competitor to commonly-used frameworks.

33 pages, 10 figures, 2 tables

References in corpus (7)

• Practical Bayesian Optimization of Machine Learning Algorithms
• A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
• Scalable Bayesian Optimization Using Deep Neural Networks
• A Tutorial on Bayesian Optimization
• The ALAMO approach to machine learning
• Scalable Global Optimization via Local Bayesian Optimization
• BOCK : Bayesian Optimization with Cylindrical Kernels

Cited by in corpus (2)

• Maximizing information from chemical engineering data sets: Applications to machine learning
• Multi-Objective Constrained Optimization for Energy Applications via Tree Ensembles