Leveraging Memory Effects and Gradient Information in Consensus-Based Optimization: On Global Convergence in Mean-Field Law

Abstract

In this paper we study consensus-based optimization (CBO), a versatile, flexible and customizable optimization method suitable for performing nonconvex and nonsmooth global optimizations in high dimensions. CBO is a multi-particle metaheuristic, which is effective in various applications and at the same time amenable to theoretical analysis thanks to its minimalistic design. The underlying dynamics, however, is flexible enough to incorporate different mechanisms widely used in evolutionary computation and machine learning, as we show by analyzing a variant of CBO which makes use of memory effects and gradient information. We rigorously prove that this dynamics converges to a global minimizer of the objective function in mean-field law for a vast class of functions under minimal assumptions on the initialization of the method. The proof in particular reveals how to leverage further, in some applications advantageous, forces in the dynamics without loosing provable global convergence. To demonstrate the benefit of the herein investigated memory effects and gradient information in certain applications, we present numerical evidence for the superiority of this CBO variant in applications such as machine learning and compressed sensing, which en passant widen the scope of applications of CBO.

35 pages, 6 figures

References in corpus (7)

• On the Global Convergence of Particle Swarm Optimization Methods
• On the mean-field limit for the consensus-based optimization
• Consensus based optimization via jump-diffusion stochastic differential equations
• Convergence of Anisotropic Consensus-Based Optimization in Mean-Field Law
• FedCBO: Reaching Group Consensus in Clustered Federated Learning through Consensus-based Optimization
• A consensus-based global optimization method with adaptive momentum estimation
• Consensus-Based Optimization for Multi-Objective Problems: A Multi-Swarm Approach

Cited by in corpus (7)

• On the Global Convergence of Particle Swarm Optimization Methods
• Consensus-Based Optimization Methods Converge Globally
• Consensus-Based Optimization for Saddle Point Problems
• CBX: Python and Julia packages for consensus-based interacting particle methods
• Consensus-Based Optimization with Truncated Noise
• MirrorCBO: A consensus-based optimization method in the spirit of mirror descent
• Defending Against Diverse Attacks in Federated Learning Through Consensus-Based Bi-Level Optimization