LibAUC: A Deep Learning Library for X-Risk Optimization

Abstract

This paper introduces the award-winning deep learning (DL) library called LibAUC for implementing state-of-the-art algorithms towards optimizing a family of risk functions named X-risks. X-risks refer to a family of compositional functions in which the loss function of each data point is defined in a way that contrasts the data point with a large number of others. They have broad applications in AI for solving classical and emerging problems, including but not limited to classification for imbalanced data (CID), learning to rank (LTR), and contrastive learning of representations (CLR). The motivation of developing LibAUC is to address the convergence issues of existing libraries for solving these problems. In particular, existing libraries may not converge or require very large mini-batch sizes in order to attain good performance for these problems, due to the usage of the standard mini-batch technique in the empirical risk minimization (ERM) framework. Our library is for deep X-risk optimization (DXO) that has achieved great success in solving a variety of tasks for CID, LTR and CLR. The contributions of this paper include: (1) It introduces a new mini-batch based pipeline for implementing DXO algorithms, which differs from existing DL pipeline in the design of controlled data samplers and dynamic mini-batch losses; (2) It provides extensive benchmarking experiments for ablation studies and comparison with existing libraries. The LibAUC library features scalable performance for millions of items to be contrasted, faster and better convergence than existing libraries for optimizing X-risks, seamless PyTorch deployment and versatile APIs for various loss optimization. Our library is available to the open source community at https://github.com/Optimization-AI/LibAUC, to facilitate further academic research and industrial applications.

arXiv commentAccepted by KDD2023

References in corpus (23)

• Adam: A Method for Stochastic Optimization
• Decoupled Weight Decay Regularization
• A Simple Framework for Contrastive Learning of Visual Representations
• Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networks
• VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning
• DeeperGCN: All You Need to Train Deeper GCNs
• TF-Ranking: Scalable TensorFlow Library for Learning-to-Rank
• Optimizing Rank-based Metrics with Blackbox Differentiation
• DIG: A Turnkey Library for Diving into Graph Deep Learning Research
• Stochastic AUC Maximization with Deep Neural Networks
• NeuralNDCG: Direct Optimisation of a Ranking Metric via Differentiable Relaxation of Sorting
• When AUC meets DRO: Optimizing Partial AUC for Deep Learning with Non-Convex Convergence Guarantee
• Benchmarking Deep AUROC Optimization: Loss Functions and Algorithmic Choices
• Large-scale Stochastic Optimization of NDCG Surrogates for Deep Learning with Provable Convergence
• Advanced Graph and Sequence Neural Networks for Molecular Property Prediction and Drug Discovery
• Multi-block-Single-probe Variance Reduced Estimator for Coupled Compositional Optimization
• Not All Semantics are Created Equal: Contrastive Self-supervised Learning with Automatic Temperature Individualization
• Momentum Accelerates the Convergence of Stochastic AUPRC Maximization
• Performance or Trust? Why Not Both. Deep AUC Maximization with Self-Supervised Learning for COVID-19 Chest X-ray Classifications
• Algorithmic Foundations of Empirical X-risk Minimization
• Provable Multi-instance Deep AUC Maximization with Stochastic Pooling
• Finite-Sum Coupled Compositional Stochastic Optimization: Theory and Applications
• Provable Stochastic Optimization for Global Contrastive Learning: Small Batch Does Not Harm Performance