Escaping local minima with derivative-free methods: a numerical investigation

Abstract

We apply a state-of-the-art, local derivative-free solver, Py-BOBYQA, to global optimization problems, and propose an algorithmic improvement that is beneficial in this context. Our numerical findings are illustrated on a commonly-used but small-scale test set of global optimization problems and associated noisy variants, and on hyperparameter tuning for the machine learning test set MNIST. As Py-BOBYQA is a model-based trust-region method, we compare mostly (but not exclusively) with other global optimization methods for which (global) models are important, such as Bayesian optimization and response surface methods; we also consider state-of-the-art representative deterministic and stochastic codes, such as DIRECT and CMA-ES. As a heuristic for escaping local minima, we find numerically that Py-BOBYQA is competitive with global optimization solvers for all accuracy/budget regimes, in both smooth and noisy settings. In particular, Py-BOBYQA variants are best performing for smooth and multiplicative noise problems in high-accuracy regimes. As a by-product, some preliminary conclusions can be drawn on the relative performance of the global solvers we have tested with default settings.

References in corpus (4)

• Practical Bayesian Optimization of Machine Learning Algorithms
• A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
• Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization
• Black-Box Optimization in Machine Learning with Trust Region Based Derivative Free Algorithm

Cited by in corpus (16)

• DESI 2024: Reconstructing Dark Energy using Crossing Statistics with DESI DR1 BAO data
• Cosmological tests of quintessence in quantum gravity
• candl: Cosmic Microwave Background Analysis with a Differentiable Likelihood
• Cosmic sign-reversal: non-parametric reconstruction of interacting dark energy with DESI DR2
• Testing the $n_s-H_0$ scaling relation with Planck-independent CMB data
• Scalar-Tensor Gravity and DESI 2024 BAO data
• Positive neutrino masses with DESI DR2 via matter conversion to dark energy
• QFactor: A Domain-Specific Optimizer for Quantum Circuit Instantiation
• Does the fraction of dark matter diminish with early dark energy?
• Explanation of high redshift luminous galaxies from JWST by early dark energy model
• Hubble tension as a window on the gravitation of the dark matter sector: Exploration of a family of models
• Exponential quintessence with momentum coupling to dark matter
• Geoff: The Generic Optimization Framework & Frontend for Particle Accelerator Controls
• The choice of Planck CMB likelihood in cosmological analyses
• Isogeometric Shape Optimization of Multi-Tapered Coaxial Baluns Simulated by an Integral Equation Method
• Neutrino mass limits and decaying dark matter: background evolution versus perturbations