Learning Surrogate Losses

Abstract

The minimization of loss functions is the heart and soul of Machine Learning. In this paper, we propose an off-the-shelf optimization approach that can minimize virtually any non-differentiable and non-decomposable loss function (e.g. Miss-classification Rate, AUC, F1, Jaccard Index, Mathew Correlation Coefficient, etc.) seamlessly. Our strategy learns smooth relaxation versions of the true losses by approximating them through a surrogate neural network. The proposed loss networks are set-wise models which are invariant to the order of mini-batch instances. Ultimately, the surrogate losses are learned jointly with the prediction model via bilevel optimization. Empirical results on multiple datasets with diverse real-life loss functions compared with state-of-the-art baselines demonstrate the efficiency of learning surrogate losses.

References in corpus (4)

• Learning to Learn without Gradient Descent by Gradient Descent
• Optimizing F-measure: A Tale of Two Approaches
• Learning to Teach with Dynamic Loss Functions
• Training Deep Neural Networks via Direct Loss Minimization

Cited by in corpus (13)

• Meta-learning PINN loss functions
• Review: Deep Learning in Electron Microscopy
• Deep Weakly-supervised Anomaly Detection
• Distribution-aware Margin Calibration for Semantic Segmentation in Images
• Online Meta-Critic Learning for Off-Policy Actor-Critic Methods
• Learning Adaptive Loss for Robust Learning with Noisy Labels
• Optimizing Black-box Metrics with Adaptive Surrogates
• Supervised Learning: No Loss No Cry
• MetricOpt: Learning to Optimize Black-Box Evaluation Metrics
• Amortized Synthesis of Constrained Configurations Using a Differentiable Surrogate
• Stateless actor-critic for instance segmentation with high-level priors
• A Unified Framework of Surrogate Loss by Refactoring and Interpolation
• EXACT: How to Train Your Accuracy