Hybrid BYOL-ViT: Efficient approach to deal with small datasets

Abstract

Supervised learning can learn large representational spaces, which are crucial for handling difficult learning tasks. However, due to the design of the model, classical image classification approaches struggle to generalize to new problems and new situations when dealing with small datasets. In fact, supervised learning can lose the location of image features which leads to supervision collapse in very deep architectures. In this paper, we investigate how self-supervision with strong and sufficient augmentation of unlabeled data can train effectively the first layers of a neural network even better than supervised learning, with no need for millions of labeled data. The main goal is to disconnect pixel data from annotation by getting generic task-agnostic low-level features. Furthermore, we look into Vision Transformers (ViT) and show that the low-level features derived from a self-supervised architecture can improve the robustness and the overall performance of this emergent architecture. We evaluated our method on one of the smallest open-source datasets STL-10 and we obtained a significant boost of performance from 41.66% to 83.25% when inputting low-level features from a self-supervised learning architecture to the ViT instead of the raw images.

References in corpus (20)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Continuous control with deep reinforcement learning
• A Simple Framework for Contrastive Learning of Visual Representations
• Deep Residual Learning for Image Recognition
• TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Language Models are Few-Shot Learners
• Empirical Evaluation of Rectified Activations in Convolutional Network
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Don't Decay the Learning Rate, Increase the Batch Size
• Prototypical Contrastive Learning of Unsupervised Representations
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Learning with Pseudo-Ensembles
• Escaping the Big Data Paradigm with Compact Transformers
• A Comprehensive Survey on Transfer Learning
• Do Vision Transformers See Like Convolutional Neural Networks?
• A critical analysis of self-supervision, or what we can learn from a single image
• Deep Reinforcement Learning and the Deadly Triad
• CrossTransformers: spatially-aware few-shot transfer