Are we done with ImageNet?

Abstract

Yes, and no. We ask whether recent progress on the ImageNet classification benchmark continues to represent meaningful generalization, or whether the community has started to overfit to the idiosyncrasies of its labeling procedure. We therefore develop a significantly more robust procedure for collecting human annotations of the ImageNet validation set. Using these new labels, we reassess the accuracy of recently proposed ImageNet classifiers, and find their gains to be substantially smaller than those reported on the original labels. Furthermore, we find the original ImageNet labels to no longer be the best predictors of this independently-collected set, indicating that their usefulness in evaluating vision models may be nearing an end. Nevertheless, we find our annotation procedure to have largely remedied the errors in the original labels, reinforcing ImageNet as a powerful benchmark for future research in visual recognition.

All five authors contributed equally. New labels at https://github.com/google-research/reassessed-imagenet

References in corpus (12)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• A Simple Framework for Contrastive Learning of Visual Representations
• Improved Baselines with Momentum Contrastive Learning
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Do ImageNet Classifiers Generalize to ImageNet?
• Self-Supervised Learning by Cross-Modal Audio-Video Clustering
• Self-training with Noisy Student improves ImageNet classification
• What Do Compressed Deep Neural Networks Forget?
• Local Aggregation for Unsupervised Learning of Visual Embeddings
• Video Representation Learning by Dense Predictive Coding
• Compounding the Performance Improvements of Assembled Techniques in a Convolutional Neural Network
• Unsupervised Learning from Video with Deep Neural Embeddings

Cited by in corpus (46)

• MLP-Mixer: An all-MLP Architecture for Vision
• CvT: Introducing Convolutions to Vision Transformers
• All Tokens Matter: Token Labeling for Training Better Vision Transformers
• When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations
• LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference
• Toward a Perspectivist Turn in Ground Truthing for Predictive Computing
• Fixing the train-test resolution discrepancy: FixEfficientNet
• Long-Short Transformer: Efficient Transformers for Language and Vision
• Large image datasets: A pyrrhic win for computer vision?
• Refiner: Refining Self-attention for Vision Transformers
• Danish Fungi 2020 -- Not Just Another Image Recognition Dataset
• ResNet strikes back: An improved training procedure in timm
• VOLO: Vision Outlooker for Visual Recognition
• Generalized Jensen-Shannon Divergence Loss for Learning with Noisy Labels
• Global Filter Networks for Image Classification
• How Well do Feature Visualizations Support Causal Understanding of CNN Activations?
• Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Research
• Self-distillation with Batch Knowledge Ensembling Improves ImageNet Classification
• Self-Damaging Contrastive Learning
• TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?
• A Simple Probabilistic Method for Deep Classification under Input-Dependent Label Noise
• Discrete Representations Strengthen Vision Transformer Robustness
• Backward-Compatible Prediction Updates: A Probabilistic Approach
• RegionCL: Can Simple Region Swapping Contribute to Contrastive Learning?
• The Benchmark Lottery
• Collaboration of Experts: Achieving 80% Top-1 Accuracy on ImageNet with 100M FLOPs
• Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labels
• Pyramid Adversarial Training Improves ViT Performance
• Multi-label Iterated Learning for Image Classification with Label Ambiguity
• No Regret Sample Selection with Noisy Labels
• Consistency Regularization Can Improve Robustness to Label Noise
• When does loss-based prioritization fail?
• A Fast Knowledge Distillation Framework for Visual Recognition
• Divergence Optimization for Noisy Universal Domain Adaptation
• Trivial or impossible -- dichotomous data difficulty masks model differences (on ImageNet and beyond)
• A Tale Of Two Long Tails
• Why Do Better Loss Functions Lead to Less Transferable Features?
• Exploring and Improving Mobile Level Vision Transformers
• A method of supervised learning from conflicting data with hidden contexts
• Life is not black and white -- Combining Semi-Supervised Learning with fuzzy labels
• Towards A Measure Of General Machine Intelligence
• Deep Classifiers with Label Noise Modeling and Distance Awareness
• Self-Supervised Learning for Large-Scale Unsupervised Image Clustering
• How Well Do Sparse Imagenet Models Transfer?
• Disentangled Variational Information Bottleneck for Multiview Representation Learning
• Perspective: Purposeful Failure in Artificial Life and Artificial Intelligence