Optimizing Black-box Metrics with Adaptive Surrogates

Abstract

We address the problem of training models with black-box and hard-to-optimize metrics by expressing the metric as a monotonic function of a small number of easy-to-optimize surrogates. We pose the training problem as an optimization over a relaxed surrogate space, which we solve by estimating local gradients for the metric and performing inexact convex projections. We analyze gradient estimates based on finite differences and local linear interpolations, and show convergence of our approach under smoothness assumptions with respect to the surrogates. Experimental results on classification and ranking problems verify the proposal performs on par with methods that know the mathematical formulation, and adds notable value when the form of the metric is unknown.

References in corpus (7)

• Optimizing F-measure: A Tale of Two Approaches
• A Short Note on Concentration Inequalities for Random Vectors with SubGaussian Norm
• Learning Surrogate Losses
• Addressing the Loss-Metric Mismatch with Adaptive Loss Alignment
• Optimizing Non-decomposable Performance Measures: A Tale of Two Classes
• Online and Stochastic Gradient Methods for Non-decomposable Loss Functions
• Surrogate Functions for Maximizing Precision at the Top

Cited by in corpus (1)

• Optimize What You Evaluate With: A Simple Yet Effective Framework For Direct Optimization Of IR Metrics