Personalizing Pre-trained Models

Abstract

Self-supervised or weakly supervised models trained on large-scale datasets have shown sample-efficient transfer to diverse datasets in few-shot settings. We consider how upstream pretrained models can be leveraged for downstream few-shot, multilabel, and continual learning tasks. Our model CLIPPER (CLIP PERsonalized) uses image representations from CLIP, a large-scale image representation learning model trained using weak natural language supervision. We developed a technique, called Multi-label Weight Imprinting (MWI), for multi-label, continual, and few-shot learning, and CLIPPER uses MWI with image representations from CLIP. We evaluated CLIPPER on 10 single-label and 5 multi-label datasets. Our model shows robust and competitive performance, and we set new benchmarks for few-shot, multi-label, and continual learning. Our lightweight technique is also compute-efficient and enables privacy-preserving applications as the data is not sent to the upstream model for fine-tuning.

References in corpus (21)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
• A Simple Framework for Contrastive Learning of Visual Representations
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Learning Transferable Visual Models From Natural Language Supervision
• Prototypical Networks for Few-shot Learning
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Improved Baselines with Momentum Contrastive Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• On First-Order Meta-Learning Algorithms
• Semi-supervised Sequence Learning
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• On Tiny Episodic Memories in Continual Learning
• One-shot Learning with Memory-Augmented Neural Networks
• Natural Adversarial Examples
• FairFace: Face Attribute Dataset for Balanced Race, Gender, and Age
• Direction Concentration Learning: Enhancing Congruency in Machine Learning
• Wise-SrNet: A Novel Architecture for Enhancing Image Classification by Learning Spatial Resolution of Feature Maps
• Decoder Choice Network for Meta-Learning
• PAL: Intelligence Augmentation using Egocentric Visual Context Detection

Cited by in corpus (1)

• Pretrained Encoders are All You Need