On The State of Data In Computer Vision: Human Annotations Remain Indispensable for Developing Deep Learning Models

Abstract

High-quality labeled datasets play a crucial role in fueling the development of machine learning (ML), and in particular the development of deep learning (DL). However, since the emergence of the ImageNet dataset and the AlexNet model in 2012, the size of new open-source labeled vision datasets has remained roughly constant. Consequently, only a minority of publications in the computer vision community tackle supervised learning on datasets that are orders of magnitude larger than Imagenet. In this paper, we survey computer vision research domains that study the effects of such large datasets on model performance across different vision tasks. We summarize the community's current understanding of those effects, and highlight some open questions related to training with massive datasets. In particular, we tackle: (a) The largest datasets currently used in computer vision research and the interesting takeaways from training on such datasets; (b) The effectiveness of pre-training on large datasets; (c) Recent advancements and hurdles facing synthetic datasets; (d) An overview of double descent and sample non-monotonicity phenomena; and finally, (e) A brief discussion of lifelong/continual learning and how it fares compared to learning from huge labeled datasets in an offline setting. Overall, our findings are that research on optimization for deep learning focuses on perfecting the training routine and thus making DL models less data hungry, while research on synthetic datasets aims to offset the cost of data labeling. However, for the time being, acquiring non-synthetic labeled data remains indispensable to boost performance.

References in corpus (31)

• Distilling the Knowledge in a Neural Network
• Conditional Generative Adversarial Nets
• Overcoming catastrophic forgetting in neural networks
• An overview of gradient descent optimization algorithms
• mixup: Beyond Empirical Risk Minimization
• Cross-lingual Language Model Pretraining
• Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels
• The Cityscapes Dataset for Semantic Urban Scene Understanding
• BEiT: BERT Pre-Training of Image Transformers
• Generating Videos with Scene Dynamics
• Transfusion: Understanding Transfer Learning for Medical Imaging
• Deep Clustering for Unsupervised Learning of Visual Features
• Deep Learning Scaling is Predictable, Empirically
• PlaNet - Photo Geolocation with Convolutional Neural Networks
• Deep Learning is Robust to Massive Label Noise
• Xception: Deep Learning with Depthwise Separable Convolutions
• MaskGAN: Better Text Generation via Filling in the______
• Unsupervised Visual Representation Learning by Context Prediction
• High-Performance Large-Scale Image Recognition Without Normalization
• Continual Learning with Deep Generative Replay
• Playing for Data: Ground Truth from Computer Games
• Learning Representations for Automatic Colorization
• SceneNet RGB-D: 5M Photorealistic Images of Synthetic Indoor Trajectories with Ground Truth
• Tencent ML-Images: A Large-Scale Multi-Label Image Database for Visual Representation Learning
• Towards Robustness Against Natural Language Word Substitutions
• RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation
• Interpretable Adversarial Perturbation in Input Embedding Space for Text
• Understanding overfitting peaks in generalization error: Analytical risk curves for $l_2$ and $l_1$ penalized interpolation
• Scaling Robot Supervision to Hundreds of Hours with RoboTurk: Robotic Manipulation Dataset through Human Reasoning and Dexterity
• Representative & Fair Synthetic Data
• Fed-Sim: Federated Simulation for Medical Imaging