Black-Box Optimization in Machine Learning with Trust Region Based Derivative Free Algorithm
arXiv:1703.06925
Abstract
In this work, we utilize a Trust Region based Derivative Free Optimization (DFO-TR) method to directly maximize the Area Under Receiver Operating Characteristic Curve (AUC), which is a nonsmooth, noisy function. We show that AUC is a smooth function, in expectation, if the distributions of the positive and negative data points obey a jointly normal distribution. The practical performance of this algorithm is compared to three prominent Bayesian optimization methods and random search. The presented numerical results show that DFO-TR surpasses Bayesian optimization and random search on various black-box optimization problem, such as maximizing AUC and hyperparameter tuning.
References in corpus (4)
- A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
- Non-stochastic Best Arm Identification and Hyperparameter Optimization
- Convergence Rate Analysis of a Stochastic Trust Region Method via Submartingales
- Stochastic Optimization Using a Trust-Region Method and Random Models
Cited by in corpus (6)
- Escaping local minima with derivative-free methods: a numerical investigation
- Newton-Type Methods for Non-Convex Optimization Under Inexact Hessian Information
- Feature Engineering and Forecasting via Derivative-free Optimization and Ensemble of Sequence-to-sequence Networks with Applications in Renewable Energy
- HyperNOMAD: Hyperparameter optimization of deep neural networks using mesh adaptive direct search
- Scalable Derivative-Free Optimization for Nonlinear Least-Squares Problems
- Efficient Hyperparameter Tuning with Dynamic Accuracy Derivative-Free Optimization