Scalable Learning of Non-Decomposable Objectives

Abstract

Modern retrieval systems are often driven by an underlying machine learning model. The goal of such systems is to identify and possibly rank the few most relevant items for a given query or context. Thus, such systems are typically evaluated using a ranking-based performance metric such as the area under the precision-recall curve, the $F_{\beta }$ score, precision at fixed recall, etc. Obviously, it is desirable to train such systems to optimize the metric of interest. In practice, due to the scalability limitations of existing approaches for optimizing such objectives, large-scale retrieval systems are instead trained to maximize classification accuracy, in the hope that performance as measured via the true objective will also be favorable. In this work we present a unified framework that, using straightforward building block bounds, allows for highly scalable optimization of a wide range of ranking-based objectives. We demonstrate the advantage of our approach on several real-life retrieval problems that are significantly larger than those considered in the literature, while achieving substantial improvement in performance over the accuracy-objective baseline.

References in corpus (1)

• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems

Cited by in corpus (17)

• The Real-World-Weight Cross-Entropy Loss Function: Modeling the Costs of Mislabeling
• Clinically Accurate Chest X-Ray Report Generation
• Robust Optimization for Fairness with Noisy Protected Groups
• Deep Weakly-supervised Anomaly Detection
• Text-Conditioned Transformer for Automatic Pronunciation Error Detection
• Wasserstein Fair Classification
• Optimizing Two-way Partial AUC with an End-to-end Framework
• FNNC: Achieving Fairness through Neural Networks
• Stochastic Optimization of Areas Under Precision-Recall Curves with Provable Convergence
• sigmoidF1: A Smooth F1 Score Surrogate Loss for Multilabel Classification
• ETH-DS3Lab at SemEval-2018 Task 7: Effectively Combining Recurrent and Convolutional Neural Networks for Relation Classification and Extraction
• Momentum Accelerates the Convergence of Stochastic AUPRC Maximization
• Optimizing Black-box Metrics with Iterative Example Weighting
• Minimizing Close-k Aggregate Loss Improves Classification
• Class-dependent Compression of Deep Neural Networks
• Using holistic event information in the trigger
• Stateless actor-critic for instance segmentation with high-level priors