ImageNet-21K Pretraining for the Masses
arXiv:2104.10972
Abstract
ImageNet-1K serves as the primary dataset for pretraining deep learning models for computer vision tasks. ImageNet-21K dataset, which is bigger and more diverse, is used less frequently for pretraining, mainly due to its complexity, low accessibility, and underestimation of its added value. This paper aims to close this gap, and make high-quality efficient pretraining on ImageNet-21K available for everyone. Via a dedicated preprocessing stage, utilization of WordNet hierarchical structure, and a novel training scheme called semantic softmax, we show that various models significantly benefit from ImageNet-21K pretraining on numerous datasets and tasks, including small mobile-oriented models. We also show that we outperform previous ImageNet-21K pretraining schemes for prominent new models like ViT and Mixer. Our proposed pretraining pipeline is efficient, accessible, and leads to SoTA reproducible results, from a publicly available dataset. The training code and pretrained models are available at: https://github.com/Alibaba-MIIL/ImageNet21K
arXiv commentAccepted to NeurIPS 2021 (Datasets and Benchmarks)
References in corpus (25)
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Decoupled Weight Decay Regularization
- mixup: Beyond Empirical Risk Minimization
- Knowledge Distillation: A Survey
- The Kinetics Human Action Video Dataset
- Improved Regularization of Convolutional Neural Networks with Cutout
- Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
- Do Deep Nets Really Need to be Deep?
- MLP-Mixer: An all-MLP Architecture for Vision
- fastai: A Layered API for Deep Learning
- A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay
- Multi-Task Learning with Deep Neural Networks: A Survey
- What makes ImageNet good for transfer learning?
- CPTR: Full Transformer Network for Image Captioning
- Gradient Surgery for Multi-Task Learning
- Adapting Auxiliary Losses Using Gradient Similarity
- Domain Adaptive Transfer Learning with Specialist Models
- FoodX-251: A Dataset for Fine-grained Food Classification
- Asymmetric Loss For Multi-Label Classification
- An Image is Worth 16x16 Words, What is a Video Worth?
- Compounding the Performance Improvements of Assembled Techniques in a Convolutional Neural Network
- Scale out for large minibatch SGD: Residual network training on ImageNet-1K with improved accuracy and reduced time to train
- Deep Networks With Large Output Spaces
- Scalable Transfer Learning with Expert Models
- Deep Ensembles for Low-Data Transfer Learning