DeepEMD: Differentiable Earth Mover's Distance for Few-Shot Learning

Abstract

In this work, we develop methods for few-shot image classification from a new perspective of optimal matching between image regions. We employ the Earth Mover's Distance (EMD) as a metric to compute a structural distance between dense image representations to determine image relevance. The EMD generates the optimal matching flows between structural elements that have the minimum matching cost, which is used to calculate the image distance for classification. To generate the important weights of elements in the EMD formulation, we design a cross-reference mechanism, which can effectively alleviate the adverse impact caused by the cluttered background and large intra-class appearance variations. To implement k-shot classification, we propose to learn a structured fully connected layer that can directly classify dense image representations with the EMD. Based on the implicit function theorem, the EMD can be inserted as a layer into the network for end-to-end training. Our extensive experiments validate the effectiveness of our algorithm which outperforms state-of-the-art methods by a significant margin on five widely used few-shot classification benchmarks, namely, miniImageNet, tieredImageNet, Fewshot-CIFAR100 (FC100), Caltech-UCSD Birds-200-2011 (CUB), and CIFAR-FewShot (CIFAR-FS). We also demonstrate the effectiveness of our method on the image retrieval task in our experiments.

DeepEMD V2, accepted by TPAMI

References in corpus (17)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
• Learning to Propagate Labels: Transductive Propagation Network for Few-shot Learning
• Delta-encoder: an effective sample synthesis method for few-shot object recognition
• Learning to Self-Train for Semi-Supervised Few-Shot Classification
• Meta-Learning with Implicit Gradients
• Differentiable Convex Optimization Layers
• On Differentiating Parameterized Argmin and Argmax Problems with Application to Bi-level Optimization
• Deep Meta-Learning: Learning to Learn in the Concept Space
• Finding Task-Relevant Features for Few-Shot Learning by Category Traversal
• Meta-Learning Update Rules for Unsupervised Representation Learning
• Generative Adversarial Residual Pairwise Networks for One Shot Learning
• Adapted Deep Embeddings: A Synthesis of Methods for $k$-Shot Inductive Transfer Learning
• On the Differentiability of the Solution to Convex Optimization Problems
• Conditional Gaussian Distribution Learning for Open Set Recognition
• CRNet: Cross-Reference Networks for Few-Shot Segmentation
• Learning from Adversarial Features for Few-Shot Classification

Cited by in corpus (11)

• An Ensemble of Epoch-wise Empirical Bayes for Few-shot Learning
• CycleSegNet: Object Co-segmentation with Cycle Refinement and Region Correspondence
• Few-Shot Incremental Learning with Continually Evolved Classifiers
• Learning to Affiliate: Mutual Centralized Learning for Few-shot Classification
• Machine learning with limited data
• Self-Point-Flow: Self-Supervised Scene Flow Estimation from Point Clouds with Optimal Transport and Random Walk
• Hierarchical Representation based Query-Specific Prototypical Network for Few-Shot Image Classification
• Will Multi-modal Data Improves Few-shot Learning?
• A${}^2$M${}^2$-Net: Adaptively Aligned Multi-Scale Moment for Few-Shot Action Recognition
• Anti-aliasing Semantic Reconstruction for Few-Shot Semantic Segmentation
• Calibrating Class Activation Maps for Long-Tailed Visual Recognition