Momentum Accelerates the Convergence of Stochastic AUPRC Maximization

Abstract

In this paper, we study stochastic optimization of areas under precision-recall curves (AUPRC), which is widely used for combating imbalanced classification tasks. Although a few methods have been proposed for maximizing AUPRC, stochastic optimization of AUPRC with convergence guarantee remains an undeveloped territory. A state-of-the-art complexity is $O(1/{\epsilon }^5)$ for finding an $\epsilon$-stationary solution. In this paper, we further improve the stochastic optimization of AURPC by (i) developing novel stochastic momentum methods with a better iteration complexity of $O(1/{\epsilon }^4)$ for finding an $\epsilon$-stationary solution; and (ii) designing a novel family of stochastic adaptive methods with the same iteration complexity, which enjoy faster convergence in practice. To this end, we propose two innovative techniques that are critical for improving the convergence: (i) the biased estimators for tracking individual ranking scores are updated in a randomized coordinate-wise manner; and (ii) a momentum update is used on top of the stochastic gradient estimator for tracking the gradient of the objective. The novel analysis of Adam-style updates is also one main contribution. Extensive experiments on various data sets demonstrate the effectiveness of the proposed algorithms. Of independent interest, the proposed stochastic momentum and adaptive algorithms are also applicable to a class of two-level stochastic dependent compositional optimization problems.

This work has been accepted by AISTATS'22

References in corpus (11)

• On the Convergence of Adam and Beyond
• Adaptive Gradient Methods with Dynamic Bound of Learning Rate
• Optimizing Rank-based Metrics with Blackbox Differentiation
• Unachievable Region in Precision-Recall Space and Its Effect on Empirical Evaluation
• Scalable Learning of Non-Decomposable Objectives
• A Ranking-based, Balanced Loss Function Unifying Classification and Localisation in Object Detection
• Accelerating Stochastic Composition Optimization
• Accelerated Method for Stochastic Composition Optimization with Nonsmooth Regularization
• Communication-Efficient Distributed Stochastic AUC Maximization with Deep Neural Networks
• A Stochastic Composite Gradient Method with Incremental Variance Reduction
• Stochastic Optimization of Areas Under Precision-Recall Curves with Provable Convergence

Cited by in corpus (1)

• LibAUC: A Deep Learning Library for X-Risk Optimization