A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark

Abstract

Representation learning promises to unlock deep learning for the long tail of vision tasks without expensive labelled datasets. Yet, the absence of a unified evaluation for general visual representations hinders progress. Popular protocols are often too constrained (linear classification), limited in diversity (ImageNet, CIFAR, Pascal-VOC), or only weakly related to representation quality (ELBO, reconstruction error). We present the Visual Task Adaptation Benchmark (VTAB), which defines good representations as those that adapt to diverse, unseen tasks with few examples. With VTAB, we conduct a large-scale study of many popular publicly-available representation learning algorithms. We carefully control confounders such as architecture and tuning budget. We address questions like: How effective are ImageNet representations beyond standard natural datasets? How do representations trained via generative and discriminative models compare? To what extent can self-supervision replace labels? And, how close are we to general visual representations?

References in corpus (16)

• Auto-Encoding Variational Bayes
• Decoupled Weight Decay Regularization
• Playing Atari with Deep Reinforcement Learning
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Deep Visual Domain Adaptation: A Survey
• Learning deep representations by mutual information estimation and maximization
• Semi-Supervised Learning with Deep Generative Models
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
• AutoAugment: Learning Augmentation Policies from Data
• Recent Advances in Autoencoder-Based Representation Learning
• What makes ImageNet good for transfer learning?
• On Mutual Information Maximization for Representation Learning
• Assessing Generative Models via Precision and Recall
• Do CIFAR-10 Classifiers Generalize to CIFAR-10?

Cited by in corpus (48)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Learning Transferable Visual Models From Natural Language Supervision
• A Survey on Visual Transformer
• Bootstrap your own latent: A new approach to self-supervised Learning
• MLP-Mixer: An all-MLP Architecture for Vision
• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
• Reproducible scaling laws for contrastive language-image learning
• Current Trends in Deep Learning for Earth Observation: An Open-source Benchmark Arena for Image Classification
• Self-supervised remote sensing feature learning: Learning Paradigms, Challenges, and Future Works
• A Survey of Deep Learning for Scientific Discovery
• How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers
• Automatic Shortcut Removal for Self-Supervised Representation Learning
• LogME: Practical Assessment of Pre-trained Models for Transfer Learning
• Exploring the Limits of Large Scale Pre-training
• Scaling Vision with Sparse Mixture of Experts
• Improving out-of-distribution generalization via multi-task self-supervised pretraining
• Assaying Out-Of-Distribution Generalization in Transfer Learning
• Scalable Transfer Learning with Expert Models
• Supervised Transfer Learning at Scale for Medical Imaging
• Effect of Pre-Training Scale on Intra- and Inter-Domain Full and Few-Shot Transfer Learning for Natural and Medical X-Ray Chest Images
• Using a thousand optimization tasks to learn hyperparameter search strategies
• Selecting Relevant Features from a Multi-domain Representation for Few-shot Classification
• Ranking and Tuning Pre-trained Models: A New Paradigm for Exploiting Model Hubs
• How Useful is Self-Supervised Pretraining for Visual Tasks?
• Learning Curves for Analysis of Deep Networks
• Deep Ensembles for Low-Data Transfer Learning
• Comparing Transfer and Meta Learning Approaches on a Unified Few-Shot Classification Benchmark
• The Benchmark Lottery
• Making Intelligence: Ethical Values in IQ and ML Benchmarks
• A comprehensive and easy-to-use multi-domain multi-task medical imaging meta-dataset
• Towards a Hypothesis on Visual Transformation based Self-Supervision
• Embedding Adaptation is Still Needed for Few-Shot Learning
• Parameter Prediction for Unseen Deep Architectures
• On Convergence of Nearest Neighbor Classifiers over Feature Transformations
• Interpretable Meta-Measure for Model Performance
• PolyViT: Co-training Vision Transformers on Images, Videos and Audio
• Ranking Neural Checkpoints
• A Unified Mixture-View Framework for Unsupervised Representation Learning
• Don't miss the Mismatch: Investigating the Objective Function Mismatch for Unsupervised Representation Learning
• Practical Transferability Estimation for Image Classification Tasks
• Memory Efficient Meta-Learning with Large Images
• One to Transfer All: A Universal Transfer Framework for Vision Foundation Model with Few Data
• ESCT3D: Efficient and Selectively Controllable Text-Driven 3D Content Generation with Gaussian Splatting
• Self-supervised Benchmark Lottery on ImageNet: Do Marginal Improvements Translate to Improvements on Similar Datasets?
• Graphs for deep learning representations
• Learning to See by Looking at Noise
• Graphs as Tools to Improve Deep Learning Methods
• Meta-Learning to Improve Pre-Training