Hyper-parameter estimation method with particle swarm optimization
arXiv:2011.11944
Abstract
Particle swarm optimization (PSO) method cannot be directly used in the problem of hyper-parameter estimation since the mathematical formulation of the mapping from hyper-parameters to loss function or generalization accuracy is unclear. Bayesian optimization (BO) framework is capable of converting the optimization of the hyper-parameters into the optimization of an acquisition function. The acquisition function is non-convex and multi-peak. So the problem can be better solved by the PSO. The proposed method in this paper uses the particle swarm method to optimize the acquisition function in the BO framework to get better hyper-parameters. The performances of proposed method in both of the classification and regression models are evaluated and demonstrated. The results on several benchmark problems are improved.
References in corpus (5)
- Practical Bayesian Optimization of Machine Learning Algorithms
- A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
- A Tutorial on Bayesian Optimization
- Dealing with Integer-valued Variables in Bayesian Optimization with Gaussian Processes
- Dealing with Categorical and Integer-valued Variables in Bayesian Optimization with Gaussian Processes