Consensus-Based Optimization Methods Converge Globally

Abstract

In this paper, we study consensus-based optimization (CBO), which is a multi-agent metaheuristic derivative-free optimization method that can globally minimize nonconvex nonsmooth functions and is amenable to theoretical analysis. Based on an experimentally supported intuition that, on average, CBO performs a gradient descent of the squared Euclidean distance to the global minimizer, we devise a novel technique for proving the convergence to the global minimizer in mean-field law for a rich class of objective functions. The result unveils internal mechanisms of CBO that are responsible for the success of the method. In particular, we prove that CBO performs a convexification of a large class of optimization problems as the number of optimizing agents goes to infinity. Furthermore, we improve prior analyses by requiring mild assumptions about the initialization of the method and by covering objectives that are merely locally Lipschitz continuous. As a core component of this analysis, we establish a quantitative nonasymptotic Laplace principle, which may be of independent interest. From the result of CBO convergence in mean-field law, it becomes apparent that the hardness of any global optimization problem is necessarily encoded in the rate of the mean-field approximation, for which we provide a novel probabilistic quantitative estimate. The combination of these results allows to obtain probabilistic global convergence guarantees of the numerical CBO method.

42 pages, 3 figures

References in corpus (10)

• A Mean Field View of the Landscape of Two-Layers Neural Networks
• A consensus-based model for global optimization and its mean-field limit
• Propagation of chaos: a review of models, methods and applications. II. Applications
• On the Global Convergence of Particle Swarm Optimization Methods
• Trainability and Accuracy of Neural Networks: An Interacting Particle System Approach
• Consensus-Based Optimization on Hypersurfaces: Well-Posedness and Mean-Field Limit
• On the mean-field limit for the consensus-based optimization
• Consensus based optimization via jump-diffusion stochastic differential equations
• Convergence of Anisotropic Consensus-Based Optimization in Mean-Field Law
• Leveraging Memory Effects and Gradient Information in Consensus-Based Optimization: On Global Convergence in Mean-Field Law

Cited by in corpus (9)

• Mean-field limits for Consensus-Based Optimization and Sampling
• Self-interacting CBO: Existence, uniqueness, and long-time convergence
• A multiscale Consensus-Based algorithm for multi-level optimization
• MirrorCBO: A consensus-based optimization method in the spirit of mirror descent
• Defending Against Diverse Attacks in Federated Learning Through Consensus-Based Bi-Level Optimization
• Uniform-in-time propagation of chaos for the Cucker--Smale model
• Swarm-based optimization with jumps: a kinetic BGK framework and convergence analysis
• Consensus-based algorithms for stochastic optimization problems
• Well-posedness and propagation of chaos for multi-agent models with strategies and diffusive effects