BOIL: Towards Representation Change for Few-shot Learning

Abstract

Model Agnostic Meta-Learning (MAML) is one of the most representative of gradient-based meta-learning algorithms. MAML learns new tasks with a few data samples using inner updates from a meta-initialization point and learns the meta-initialization parameters with outer updates. It has recently been hypothesized that representation reuse, which makes little change in efficient representations, is the dominant factor in the performance of the meta-initialized model through MAML in contrast to representation change, which causes a significant change in representations. In this study, we investigate the necessity of representation change for the ultimate goal of few-shot learning, which is solving domain-agnostic tasks. To this aim, we propose a novel meta-learning algorithm, called BOIL (Body Only update in Inner Loop), which updates only the body (extractor) of the model and freezes the head (classifier) during inner loop updates. BOIL leverages representation change rather than representation reuse. This is because feature vectors (representations) have to move quickly to their corresponding frozen head vectors. We visualize this property using cosine similarity, CKA, and empirical results without the head. BOIL empirically shows significant performance improvement over MAML, particularly on cross-domain tasks. The results imply that representation change in gradient-based meta-learning approaches is a critical component.

24 pages, 26 figures, 19 tables, ICLR 2021 published

References in corpus (13)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Prototypical Networks for Few-shot Learning
• Empirical Evaluation of Rectified Activations in Convolutional Network
• Wide Residual Networks
• Fine-Grained Visual Classification of Aircraft
• On First-Order Meta-Learning Algorithms
• Similarity of Neural Network Representations Revisited
• A Closer Look at Few-shot Classification
• Fast Context Adaptation via Meta-Learning
• Few-Shot Learning with Metric-Agnostic Conditional Embeddings
• Cross-Domain Few-Shot Classification via Learned Feature-Wise Transformation
• Meta-Learning Requires Meta-Augmentation
• Cross-Domain Few-Shot Learning with Meta Fine-Tuning

Cited by in corpus (13)

• FedBABU: Towards Enhanced Representation for Federated Image Classification
• Enhancing Few-Shot Image Classification through Learnable Multi-Scale Embedding and Attention Mechanisms
• ReFine: Re-randomization before Fine-tuning for Cross-domain Few-shot Learning
• Improving Generalization in Meta-learning via Task Augmentation
• p-Meta: Towards On-device Deep Model Adaptation
• Meta-Learning with Fewer Tasks through Task Interpolation
• Dizygotic Conditional Variational AutoEncoder for Multi-Modal and Partial Modality Absent Few-Shot Learning
• Multi-Level Correlation Network For Few-Shot Image Classification
• A Channel Coding Benchmark for Meta-Learning
• Bilevel Optimization for Machine Learning: Algorithm Design and Convergence Analysis
• Model-Agnostic Learning to Meta-Learn
• MDFM: Multi-Decision Fusing Model for Few-Shot Learning
• Meta-Learning with Task-Adaptive Loss Function for Few-Shot Learning