Revisiting Unreasonable Effectiveness of Data in Deep Learning Era

Abstract

The success of deep learning in vision can be attributed to: (a) models with high capacity; (b) increased computational power; and (c) availability of large-scale labeled data. Since 2012, there have been significant advances in representation capabilities of the models and computational capabilities of GPUs. But the size of the biggest dataset has surprisingly remained constant. What will happen if we increase the dataset size by 10x or 100x? This paper takes a step towards clearing the clouds of mystery surrounding the relationship between `enormous data' and visual deep learning. By exploiting the JFT-300M dataset which has more than 375M noisy labels for 300M images, we investigate how the performance of current vision tasks would change if this data was used for representation learning. Our paper delivers some surprising (and some expected) findings. First, we find that the performance on vision tasks increases logarithmically based on volume of training data size. Second, we show that representation learning (or pre-training) still holds a lot of promise. One can improve performance on many vision tasks by just training a better base model. Finally, as expected, we present new state-of-the-art results for different vision tasks including image classification, object detection, semantic segmentation and human pose estimation. Our sincere hope is that this inspires vision community to not undervalue the data and develop collective efforts in building larger datasets.

arXiv commentICCV 2017 camera ready

References in corpus (18)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• Generating Videos with Scene Dynamics
• Adversarial Feature Learning
• Rich feature hierarchies for accurate object detection and semantic segmentation
• PlaNet - Photo Geolocation with Convolutional Neural Networks
• Xception: Deep Learning with Depthwise Separable Convolutions
• What makes ImageNet good for transfer learning?
• Unsupervised Visual Representation Learning by Context Prediction
• Unsupervised Learning of Visual Representations using Videos
• Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields
• Towards Accurate Multi-person Pose Estimation in the Wild
• Analyzing the Performance of Multilayer Neural Networks for Object Recognition
• Webly Supervised Learning of Convolutional Networks
• Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hours
• Large-Scale Deep Learning on the YFCC100M Dataset
• The Unreasonable Effectiveness of Noisy Data for Fine-Grained Recognition
• Learning Visual Features from Large Weakly Supervised Data

Cited by in corpus (68)

• Attention Mechanisms in Computer Vision: A Survey
• Snorkel: Rapid Training Data Creation with Weak Supervision
• Data Augmentation using Random Image Cropping and Patching for Deep CNNs
• The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale
• WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning
• Light-Weight RefineNet for Real-Time Semantic Segmentation
• Materials science in the era of large language models: a perspective
• Rethinking ImageNet Pre-training
• Domain Adaptive Transfer Learning with Specialist Models
• Development and Validation of Deep Learning Algorithms for Detection of Critical Findings in Head CT Scans
• Rethinking the Hyperparameters for Fine-tuning
• Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling
• Ten Quick Tips for Deep Learning in Biology
• A Better Baseline for AVA
• Data Distillation: Towards Omni-Supervised Learning
• COCO-Stuff: Thing and Stuff Classes in Context
• Scale out for large minibatch SGD: Residual network training on ImageNet-1K with improved accuracy and reduced time to train
• Support-set bottlenecks for video-text representation learning
• Classifying Exoplanet Candidates with Convolutional Neural Networks: Application to the Next Generation Transit Survey
• MaskLab: Instance Segmentation by Refining Object Detection with Semantic and Direction Features
• Rethinking Self-driving: Multi-task Knowledge for Better Generalization and Accident Explanation Ability
• Zero-shot Knowledge Transfer via Adversarial Belief Matching
• Zero-shot Recognition via Semantic Embeddings and Knowledge Graphs
• Memory Based Online Learning of Deep Representations from Video Streams
• Iterative Visual Reasoning Beyond Convolutions
• CEREALS - Cost-Effective REgion-based Active Learning for Semantic Segmentation
• Snorkel DryBell: A Case Study in Deploying Weak Supervision at Industrial Scale
• Quantifying the Performance of Federated Transfer Learning
• Addressing Missing Labels in Large-Scale Sound Event Recognition Using a Teacher-Student Framework With Loss Masking
• Knowledge Concentration: Learning 100K Object Classifiers in a Single CNN
• Automatic localization and decoding of honeybee markers using deep convolutional neural networks
• Temporal Dynamic Graph LSTM for Action-driven Video Object Detection
• Applying Cooperative Machine Learning to Speed Up the Annotation of Social Signals in Large Multi-modal Corpora
• Fix your classifier: the marginal value of training the last weight layer
• DNN or k-NN: That is the Generalize vs. Memorize Question
• KINN: Incorporating Expert Knowledge in Neural Networks
• Share your Model instead of your Data: Privacy Preserving Mimic Learning for Ranking
• Tree-structured Kronecker Convolutional Network for Semantic Segmentation
• Active learning using weakly supervised signals for quality inspection
• Image Enhanced Rotation Prediction for Self-Supervised Learning
• Unsupervised Vision-and-Language Pre-training Without Parallel Images and Captions
• Stochastic Downsampling for Cost-Adjustable Inference and Improved Regularization in Convolutional Networks
• Sparsely Aggregated Convolutional Networks
• Robust and On-the-fly Dataset Denoising for Image Classification
• The Resistance to Label Noise in K-NN and DNN Depends on its Concentration
• Semi-Supervised Noisy Student Pre-training on EfficientNet Architectures for Plant Pathology Classification
• An empirical study of pretrained representations for few-shot classification
• Large-Scale Attribute-Object Compositions
• Question Generation from SQL Queries Improves Neural Semantic Parsing
• To Pretrain or Not to Pretrain: Examining the Benefits of Pretraining on Resource Rich Tasks
• Can We Achieve More with Less? Exploring Data Augmentation for Toxic Comment Classification
• Impact of Training Dataset Size on Neural Answer Selection Models
• When Homomorphic Cryptosystem Meets Differential Privacy: Training Machine Learning Classifier with Privacy Protection
• Improving image generative models with human interactions
• Unsupervised Multi-label Dataset Generation from Web Data
• Weakly- and Semi-Supervised Panoptic Segmentation
• Designing for the Long Tail of Machine Learning
• Classification of sparsely labeled spatio-temporal data through semi-supervised adversarial learning
• Dynamic Graph Correlation Learning for Disease Diagnosis with Incomplete Labels
• Top squark signal significance enhancement by different Machine Learning Algorithms
• A deep neural network for multi-species fish detection using multiple acoustic cameras
• Using mixup as regularization and tuning hyper-parameters for ResNets
• Challenging Images For Minds and Machines
• Detecting Fetal Alcohol Spectrum Disorder in children using Artificial Neural Network
• GAIA: A Transfer Learning System of Object Detection that Fits Your Needs
• Mitigating Generation Shifts for Generalized Zero-Shot Learning
• Function space analysis of deep learning representation layers
• Parameter Reference Loss for Unsupervised Domain Adaptation