Characterizing Generalization under Out-Of-Distribution Shifts in Deep Metric Learning
arXiv:2107.09562
Abstract
Deep Metric Learning (DML) aims to find representations suitable for zero-shot transfer to a priori unknown test distributions. However, common evaluation protocols only test a single, fixed data split in which train and test classes are assigned randomly. More realistic evaluations should consider a broad spectrum of distribution shifts with potentially varying degree and difficulty. In this work, we systematically construct train-test splits of increasing difficulty and present the ooDML benchmark to characterize generalization under out-of-distribution shifts in DML. ooDML is designed to probe the generalization performance on much more challenging, diverse train-to-test distribution shifts. Based on our new benchmark, we conduct a thorough empirical analysis of state-of-the-art DML methods. We find that while generalization tends to consistently degrade with difficulty, some methods are better at retaining performance as the distribution shift increases. Finally, we propose few-shot DML as an efficient way to consistently improve generalization in response to unknown test shifts presented in ooDML. Code available here: https://github.com/CompVis/Characterizing_Generalization_in_DML.
arXiv comment35th Conference on Neural Information Processing Systems (NeurIPS 2021)
References in corpus (14)
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- Learning Transferable Visual Models From Natural Language Supervision
- Prototypical Networks for Few-shot Learning
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- In Defense of the Triplet Loss for Person Re-Identification
- Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
- Do ImageNet Classifiers Generalize to ImageNet?
- Out-of-Distribution Generalization via Risk Extrapolation (REx)
- Exploring the Landscape of Spatial Robustness
- SoftTriple Loss: Deep Metric Learning Without Triplet Sampling
- Sharing Matters for Generalization in Deep Metric Learning
- MIC: Mining Interclass Characteristics for Improved Metric Learning
- Hardness-Aware Deep Metric Learning
- S2SD: Simultaneous Similarity-based Self-Distillation for Deep Metric Learning