Self-supervised Label Augmentation via Input Transformations

Abstract

Self-supervised learning, which learns by constructing artificial labels given only the input signals, has recently gained considerable attention for learning representations with unlabeled datasets, i.e., learning without any human-annotated supervision. In this paper, we show that such a technique can be used to significantly improve the model accuracy even under fully-labeled datasets. Our scheme trains the model to learn both original and self-supervised tasks, but is different from conventional multi-task learning frameworks that optimize the summation of their corresponding losses. Our main idea is to learn a single unified task with respect to the joint distribution of the original and self-supervised labels, i.e., we augment original labels via self-supervision of input transformation. This simple, yet effective approach allows to train models easier by relaxing a certain invariant constraint during learning the original and self-supervised tasks simultaneously. It also enables an aggregated inference which combines the predictions from different augmentations to improve the prediction accuracy. Furthermore, we propose a novel knowledge transfer technique, which we refer to as self-distillation, that has the effect of the aggregated inference in a single (faster) inference. We demonstrate the large accuracy improvement and wide applicability of our framework on various fully-supervised settings, e.g., the few-shot and imbalanced classification scenarios.

arXiv commentAccepted to ICML 2020. Code available at https://github.com/hankook/SLA

Cited by in corpus (13)

• CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances
• Hierarchical Self-supervised Augmented Knowledge Distillation
• ReSSL: Relational Self-Supervised Learning with Weak Augmentation
• PGL: Prior-Guided Local Self-supervised Learning for 3D Medical Image Segmentation
• Meta-Baseline: Exploring Simple Meta-Learning for Few-Shot Learning
• Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillation
• Image Enhanced Rotation Prediction for Self-Supervised Learning
• Dizygotic Conditional Variational AutoEncoder for Multi-Modal and Partial Modality Absent Few-Shot Learning
• Pareto Self-Supervised Training for Few-Shot Learning
• Task Affinity with Maximum Bipartite Matching in Few-Shot Learning
• Transform consistency for learning with noisy labels
• Self-supervision of Feature Transformation for Further Improving Supervised Learning
• Explicit homography estimation improves contrastive self-supervised learning