MetricOpt: Learning to Optimize Black-Box Evaluation Metrics

Abstract

We study the problem of directly optimizing arbitrary non-differentiable task evaluation metrics such as misclassification rate and recall. Our method, named MetricOpt, operates in a black-box setting where the computational details of the target metric are unknown. We achieve this by learning a differentiable value function, which maps compact task-specific model parameters to metric observations. The learned value function is easily pluggable into existing optimizers like SGD and Adam, and is effective for rapidly finetuning a pre-trained model. This leads to consistent improvements since the value function provides effective metric supervision during finetuning, and helps to correct the potential bias of loss-only supervision. MetricOpt achieves state-of-the-art performance on a variety of metrics for (image) classification, image retrieval and object detection. Solid benefits are found over competing methods, which often involve complex loss design or adaptation. MetricOpt also generalizes well to new tasks and model architectures.

CVPR 2021 (Oral), Supplementary Materials added

References in corpus (10)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Fast Context Adaptation via Meta-Learning
• Multi-Similarity Loss with General Pair Weighting for Deep Metric Learning
• Learning to Teach with Dynamic Loss Functions
• Training Deep Neural Networks via Direct Loss Minimization
• Understanding and correcting pathologies in the training of learned optimizers
• Learning Surrogate Losses
• Local Descriptors Optimized for Average Precision
• Metric-Optimized Example Weights