MC-SSL0.0: Towards Multi-Concept Self-Supervised Learning

Abstract

Self-supervised pretraining is the method of choice for natural language processing models and is rapidly gaining popularity in many vision tasks. Recently, self-supervised pretraining has shown to outperform supervised pretraining for many downstream vision applications, marking a milestone in the area. This superiority is attributed to the negative impact of incomplete labelling of the training images, which convey multiple concepts, but are annotated using a single dominant class label. Although Self-Supervised Learning (SSL), in principle, is free of this limitation, the choice of pretext task facilitating SSL is perpetuating this shortcoming by driving the learning process towards a single concept output. This study aims to investigate the possibility of modelling all the concepts present in an image without using labels. In this aspect the proposed SSL frame-work MC-SSL0.0 is a step towards Multi-Concept Self-Supervised Learning (MC-SSL) that goes beyond modelling single dominant label in an image to effectively utilise the information from all the concepts present in it. MC-SSL0.0 consists of two core design concepts, group masked model learning and learning of pseudo-concept for data token using a momentum encoder (teacher-student) framework. The experimental results on multi-label and multi-class image classification downstream tasks demonstrate that MC-SSL0.0 not only surpasses existing SSL methods but also outperforms supervised transfer learning. The source code will be made publicly available for community to train on bigger corpus.

References in corpus (18)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Distilling the Knowledge in a Neural Network
• mixup: Beyond Empirical Risk Minimization
• Bootstrap your own latent: A new approach to self-supervised Learning
• Gaussian Error Linear Units (GELUs)
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Learning deep representations by mutual information estimation and maximization
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Zero-Shot Text-to-Image Generation
• BEiT: BERT Pre-Training of Image Transformers
• AutoAugment: Learning Augmentation Policies from Data
• Adversarially Learned Inference
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Large Scale Adversarial Representation Learning
• Unsupervised Learning by Predicting Noise
• Masked Autoencoders Are Scalable Vision Learners
• Query2Label: A Simple Transformer Way to Multi-Label Classification
• Self-Supervised Relational Reasoning for Representation Learning