Adapted Deep Embeddings: A Synthesis of Methods for $k$-Shot Inductive Transfer Learning

Abstract

The focus in machine learning has branched beyond training classifiers on a single task to investigating how previously acquired knowledge in a source domain can be leveraged to facilitate learning in a related target domain, known as inductive transfer learning. Three active lines of research have independently explored transfer learning using neural networks. In weight transfer, a model trained on the source domain is used as an initialization point for a network to be trained on the target domain. In deep metric learning, the source domain is used to construct an embedding that captures class structure in both the source and target domains. In few-shot learning, the focus is on generalizing well in the target domain based on a limited number of labeled examples. We compare state-of-the-art methods from these three paradigms and also explore hybrid adapted-embedding methods that use limited target-domain data to fine tune embeddings constructed from source-domain data. We conduct a systematic comparison of methods in a variety of domains, varying the number of labeled instances available in the target domain ($k$), as well as the number of target-domain classes. We reach three principal conclusions: (1) Deep embeddings are far superior, compared to weight transfer, as a starting point for inter-domain transfer or model re-use (2) Our hybrid methods robustly outperform every few-shot learning and every deep metric learning method previously proposed, with a mean error reduction of 34% over state-of-the-art. (3) Among loss functions for discovering embeddings, the histogram loss (Ustinova & Lempitsky, 2016) is most robust. We hope our results will motivate a unification of research in weight transfer, deep metric learning, and few-shot learning.

Cited by in corpus (17)

• Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?
• Few-Shot Learning via Embedding Adaptation with Set-to-Set Functions
• Small Sample Learning in Big Data Era
• On Interpretability of Artificial Neural Networks: A Survey
• DeepEMD: Differentiable Earth Mover's Distance for Few-Shot Learning
• Meta-Transfer Learning for Few-Shot Learning
• Semantic Drift Compensation for Class-Incremental Learning
• Stochastic Prototype Embeddings
• Learning Metrics from Teachers: Compact Networks for Image Embedding
• Domain Generalization via Semi-supervised Meta Learning
• A Plug-in Method for Representation Factorization in Connectionist Models
• Variational Feature Disentangling for Fine-Grained Few-Shot Classification
• Neural Networks Trained on Natural Scenes Exhibit Gestalt Closure
• A Hybrid Approach with Optimization and Metric-based Meta-Learner for Few-Shot Learning
• Compositional Embeddings for Multi-Label One-Shot Learning
• Open-Ended Content-Style Recombination Via Leakage Filtering
• ProtoShotXAI: Using Prototypical Few-Shot Architecture for Explainable AI