On the Global Convergence of Particle Swarm Optimization Methods

Abstract

In this paper we provide a rigorous convergence analysis for the renowned particle swarm optimization method by using tools from stochastic calculus and the analysis of partial differential equations. Based on a time-continuous formulation of the particle dynamics as a system of stochastic differential equations, we establish convergence to a global minimizer of a possibly nonconvex and nonsmooth objective function in two steps. First, we prove consensus formation of an associated mean-field dynamics by analyzing the time-evolution of the variance of the particle distribution. We then show that this consensus is close to a global minimizer by employing the asymptotic Laplace principle and a tractability condition on the energy landscape of the objective function. These results allow for the usage of memory mechanisms, and hold for a rich class of objectives provided certain conditions of well-preparation of the hyperparameters and the initial datum. In a second step, at least for the case without memory effects, we provide a quantitative result about the mean-field approximation of particle swarm optimization, which specifies the convergence of the interacting particle system to the associated mean-field limit. Combining these two results allows for global convergence guarantees of the numerical particle swarm optimization method with provable polynomial complexity. To demonstrate the applicability of the method we propose an efficient and parallelizable implementation, which is tested in particular on a competitive and well-understood high-dimensional benchmark problem in machine learning.

28 pages, 4 figures

References in corpus (3)

• More is the Same; Phase Transitions and Mean Field Theories
• Convergence of Anisotropic Consensus-Based Optimization in Mean-Field Law
• Leveraging Memory Effects and Gradient Information in Consensus-Based Optimization: On Global Convergence in Mean-Field Law

Cited by in corpus (12)

• Consensus-Based Optimization Methods Converge Globally
• Convergence of Anisotropic Consensus-Based Optimization in Mean-Field Law
• Consensus-Based Optimization for Saddle Point Problems
• Leveraging Memory Effects and Gradient Information in Consensus-Based Optimization: On Global Convergence in Mean-Field Law
• Mean-field limits for Consensus-Based Optimization and Sampling
• Consensus based optimization with memory effects: random selection and applications
• CBX: Python and Julia packages for consensus-based interacting particle methods
• Consensus-Based Optimization with Truncated Noise
• Defending Against Diverse Attacks in Federated Learning Through Consensus-Based Bi-Level Optimization
• MirrorCBO: A consensus-based optimization method in the spirit of mirror descent
• Swarm-based optimization with jumps: a kinetic BGK framework and convergence analysis
• Micro-Macro Decomposition of Particle Swarm Optimization Methods