Do CIFAR-10 Classifiers Generalize to CIFAR-10?

Abstract

Machine learning is currently dominated by largely experimental work focused on improvements in a few key tasks. However, the impressive accuracy numbers of the best performing models are questionable because the same test sets have been used to select these models for multiple years now. To understand the danger of overfitting, we measure the accuracy of CIFAR-10 classifiers by creating a new test set of truly unseen images. Although we ensure that the new test set is as close to the original data distribution as possible, we find a large drop in accuracy (4% to 10%) for a broad range of deep learning models. Yet more recent models with higher original accuracy show a smaller drop and better overall performance, indicating that this drop is likely not due to overfitting based on adaptivity. Instead, we view our results as evidence that current accuracy numbers are brittle and susceptible to even minute natural variations in the data distribution.

References in corpus (2)

• Improving neural networks by preventing co-adaptation of feature detectors
• The Ladder: A Reliable Leaderboard for Machine Learning Competitions

Cited by in corpus (75)

• Shortcut Learning in Deep Neural Networks
• Confounding variables can degrade generalization performance of radiological deep learning models
• AutoAugment: Learning Augmentation Policies from Data
• Adversarial Examples Are Not Bugs, They Are Features
• Standardized Assessment of Automatic Segmentation of White Matter Hyperintensities and Results of the WMH Segmentation Challenge
• A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark
• Benchmarking Neural Network Robustness to Common Corruptions and Surface Variations
• The Conditional Entropy Bottleneck
• What Can ResNet Learn Efficiently, Going Beyond Kernels?
• Improving Robustness Without Sacrificing Accuracy with Patch Gaussian Augmentation
• MEMO: Test Time Robustness via Adaptation and Augmentation
• Test-Time Training with Self-Supervision for Generalization under Distribution Shifts
• Adversarial NLI: A New Benchmark for Natural Language Understanding
• Differentially Private Learning Needs Better Features (or Much More Data)
• Self-Supervised Policy Adaptation during Deployment
• Cold Case: The Lost MNIST Digits
• Seabed classification using physics-based modeling and machine learning
• ConfusionFlow: A model-agnostic visualization for temporal analysis of classifier confusion
• Batch Normalization is a Cause of Adversarial Vulnerability
• Overfitting in adversarially robust deep learning
• A Selective Overview of Deep Learning
• Exploring the Limits of Large Scale Pre-training
• sharpDARTS: Faster and More Accurate Differentiable Architecture Search
• Reproducibility in Machine Learning for Health
• Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracy
• Meta Transition Adaptation for Robust Deep Learning with Noisy Labels
• Deep regularization and direct training of the inner layers of Neural Networks with Kernel Flows
• The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuning
• Early detection of sepsis utilizing deep learning on electronic health record event sequences
• Show Your Work: Improved Reporting of Experimental Results
• HARK Side of Deep Learning -- From Grad Student Descent to Automated Machine Learning
• Image Based Identification of Ghanaian Timbers Using the XyloTron: Opportunities, Risks and Challenges
• That which we call private
• Understanding Generalization in Adversarial Training via the Bias-Variance Decomposition
• On Interaction Between Augmentations and Corruptions in Natural Corruption Robustness
• Distributionally Adversarial Attack
• Overinterpretation reveals image classification model pathologies
• SSFN -- Self Size-estimating Feed-forward Network with Low Complexity, Limited Need for Human Intervention, and Consistent Behaviour across Trials
• The Benchmark Lottery
• Identifying Statistical Bias in Dataset Replication
• Detecting Overfitting via Adversarial Examples
• Laplacian Networks: Bounding Indicator Function Smoothness for Neural Network Robustness
• DropCluster: A structured dropout for convolutional networks
• Robust neural circuit reconstruction from serial electron microscopy with convolutional recurrent networks
• Re-using Adversarial Mask Discriminators for Test-time Training under Distribution Shifts
• Deep learning for pedestrians: backpropagation in CNNs
• DeepNNK: Explaining deep models and their generalization using polytope interpolation
• Using Wavelets and Spectral Methods to Study Patterns in Image-Classification Datasets
• Principal Component Networks: Parameter Reduction Early in Training
• Supervised Domain Adaptation using Graph Embedding
• Natural Perturbed Training for General Robustness of Neural Network Classifiers
• Rip van Winkle's Razor: A Simple Estimate of Overfit to Test Data
• What Does Rotation Prediction Tell Us about Classifier Accuracy under Varying Testing Environments?
• Self-Supervised Dynamic Networks for Covariate Shift Robustness
• Out-distribution aware Self-training in an Open World Setting
• Redesigning the classification layer by randomizing the class representation vectors
• Robust, Extensible, and Fast: Teamed Classifiers for Vehicle Tracking and Vehicle Re-ID in Multi-Camera Networks
• Optimal multiclass overfitting by sequence reconstruction from Hamming queries
• High-Dimensional Optimization in Adaptive Random Subspaces
• Active Online Learning with Hidden Shifting Domains
• Towards Robust Pattern Recognition: A Review
• Learning and Inference in Imaginary Noise Models
• What do CNN neurons learn: Visualization & Clustering
• See the World through Network Cameras
• Reappraising Domain Generalization in Neural Networks
• Covariate Shift in High-Dimensional Random Feature Regression
• Wiggling Weights to Improve the Robustness of Classifiers
• An Empirical Study on Compressed Decentralized Stochastic Gradient Algorithms with Overparameterized Models
• NeuroView: Explainable Deep Network Decision Making
• Regularized joint mixture models
• Computational principles of intelligence: learning and reasoning with neural networks
• Adversarial Examples as an Input-Fault Tolerance Problem
• Towards Making Deep Transfer Learning Never Hurt
• Adversarial Data Encryption
• Seven Myths in Machine Learning Research