Will Multi-modal Data Improves Few-shot Learning?

Abstract

Most few-shot learning models utilize only one modality of data. We would like to investigate qualitatively and quantitatively how much will the model improve if we add an extra modality (i.e. text description of the image), and how it affects the learning procedure. To achieve this goal, we propose four types of fusion method to combine the image feature and text feature. To verify the effectiveness of improvement, we test the fusion methods with two classical few-shot learning models - ProtoNet and MAML, with image feature extractors such as ConvNet and ResNet12. The attention-based fusion method works best, which improves the classification accuracy by a large margin around 30% comparing to the baseline result.

arXiv commentProject Report

References in corpus (13)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Learning Transferable Visual Models From Natural Language Supervision
• Prototypical Networks for Few-shot Learning
• Deep Residual Learning for Image Recognition
• Matching Networks for One Shot Learning
• Learning to Propagate Labels: Transductive Propagation Network for Few-shot Learning
• Few-Shot Learning with Graph Neural Networks
• Simple Baseline for Visual Question Answering
• Meta-Learning with Latent Embedding Optimization
• Few-Shot Learning with Metric-Agnostic Conditional Embeddings
• DeepEMD: Differentiable Earth Mover's Distance for Few-Shot Learning
• DPGN: Distribution Propagation Graph Network for Few-shot Learning
• CentralNet: a Multilayer Approach for Multimodal Fusion