Stochastic AUC Maximization with Deep Neural Networks

Abstract

Stochastic AUC maximization has garnered an increasing interest due to better fit to imbalanced data classification. However, existing works are limited to stochastic AUC maximization with a linear predictive model, which restricts its predictive power when dealing with extremely complex data. In this paper, we consider stochastic AUC maximization problem with a deep neural network as the predictive model. Building on the saddle point reformulation of a surrogated loss of AUC, the problem can be cast into a {\it non-convex concave} min-max problem. The main contribution made in this paper is to make stochastic AUC maximization more practical for deep neural networks and big data with theoretical insights as well. In particular, we propose to explore Polyak-Łojasiewicz (PL) condition that has been proved and observed in deep learning, which enables us to develop new stochastic algorithms with even faster convergence rate and more practical step size scheme. An AdaGrad-style algorithm is also analyzed under the PL condition with adaptive convergence rate. Our experimental results demonstrate the effectiveness of the proposed algorithms.

arXiv commentAccepted by ICLR 2020

References in corpus (17)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Sequence to Sequence Learning with Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
• Non-convex Finite-Sum Optimization Via SCSG Methods
• Hybrid Block Successive Approximation for One-Sided Non-Convex Min-Max Problems: Algorithms and Applications
• A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks
• Stochastic Recursive Gradient Algorithm for Nonconvex Optimization
• A Simple Proximal Stochastic Gradient Method for Nonsmooth Nonconvex Optimization
• What is Local Optimality in Nonconvex-Nonconcave Minimax Optimization?
• An Improved Analysis of Training Over-parameterized Deep Neural Networks
• An Alternative View: When Does SGD Escape Local Minima?
• Characterization of Gradient Dominance and Regularity Conditions for Neural Networks
• Stability and Generalization of Learning Algorithms that Converge to Global Optima
• Solving Non-Convex Non-Concave Min-Max Games Under Polyak-Łojasiewicz Condition

Cited by in corpus (10)

• Optimal Epoch Stochastic Gradient Descent Ascent Methods for Min-Max Optimization
• Optimizing Two-way Partial AUC with an End-to-end Framework
• Towards Better Understanding of Adaptive Gradient Algorithms in Generative Adversarial Nets
• Eating Garlic Prevents COVID-19 Infection: Detecting Misinformation on the Arabic Content of Twitter
• Stochastic Optimization of Areas Under Precision-Recall Curves with Provable Convergence
• LibAUC: A Deep Learning Library for X-Risk Optimization
• Near Optimal Stochastic Algorithms for Finite-Sum Unbalanced Convex-Concave Minimax Optimization
• CDMA: A Practical Cross-Device Federated Learning Algorithm for General Minimax Problems
• Minimax Problems with Coupled Linear Constraints: Computational Complexity, Duality and Solution Methods
• Federated Deep AUC Maximization for Heterogeneous Data with a Constant Communication Complexity