Hybrid BYOL-ViT: Efficient approach to deal with small datasets
arXiv:2111.04845
Abstract
Supervised learning can learn large representational spaces, which are crucial for handling difficult learning tasks. However, due to the design of the model, classical image classification approaches struggle to generalize to new problems and new situations when dealing with small datasets. In fact, supervised learning can lose the location of image features which leads to supervision collapse in very deep architectures. In this paper, we investigate how self-supervision with strong and sufficient augmentation of unlabeled data can train effectively the first layers of a neural network even better than supervised learning, with no need for millions of labeled data. The main goal is to disconnect pixel data from annotation by getting generic task-agnostic low-level features. Furthermore, we look into Vision Transformers (ViT) and show that the low-level features derived from a self-supervised architecture can improve the robustness and the overall performance of this emergent architecture. We evaluated our method on one of the smallest open-source datasets STL-10 and we obtained a significant boost of performance from 41.66% to 83.25% when inputting low-level features from a self-supervised learning architecture to the ViT instead of the raw images.
References in corpus (6)
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation
- Language Models are Few-Shot Learners
- Learning with Pseudo-Ensembles
- Escaping the Big Data Paradigm with Compact Transformers
- Do Vision Transformers See Like Convolutional Neural Networks?