An Empirical Study and Analysis on Open-Set Semi-Supervised Learning

Abstract

Pseudo-labeling (PL) and Data Augmentation-based Consistency Training (DACT) are two approaches widely used in Semi-Supervised Learning (SSL) methods. These methods exhibit great power in many machine learning tasks by utilizing unlabeled data for efficient training. But in a more realistic setting (termed as open-set SSL), where unlabeled dataset contains out-of-distribution (OOD) samples, the traditional SSL methods suffer severe performance degradation. Recent approaches mitigate the negative influence of OOD samples by filtering them out from the unlabeled data. However, it is not clear whether directly removing the OOD samples is the best choice. Furthermore, why PL and DACT could perform differently in open-set SSL remains a mystery. In this paper, we thoroughly analyze various SSL methods (PL and DACT) on open-set SSL and discuss pros and cons of these two approaches separately. Based on our analysis, we propose Style Disturbance to improve traditional SSL methods on open-set SSL and experimentally show our approach can achieve state-of-the-art results on various datasets by utilizing OOD samples properly. We believe our study can bring new insights for SSL research.

References in corpus (27)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• A Simple Framework for Contrastive Learning of Visual Representations
• mixup: Beyond Empirical Risk Minimization
• Improved Regularization of Convolutional Neural Networks with Cutout
• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
• Deep Visual Domain Adaptation: A Survey
• Improved Baselines with Momentum Contrastive Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Wide Residual Networks
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction
• AutoAugment: Learning Augmentation Policies from Data
• What Makes for Good Views for Contrastive Learning?
• Transfusion: Understanding Transfer Learning for Medical Imaging
• Exploring Simple Siamese Representation Learning
• Shake-Shake regularization
• Domain Adaptation for Visual Applications: A Comprehensive Survey
• Style Augmentation: Data Augmentation via Style Randomization
• A critical analysis of self-supervision, or what we can learn from a single image
• What makes instance discrimination good for transfer learning?
• VideoMix: Rethinking Data Augmentation for Video Classification
• Contrastive Learning with Stronger Augmentations
• Content Aware Neural Style Transfer
• Clusterability as an Alternative to Anchor Points When Learning with Noisy Labels
• On Data-Augmentation and Consistency-Based Semi-Supervised Learning
• OpenCoS: Contrastive Semi-supervised Learning for Handling Open-set Unlabeled Data
• Can Pretext-Based Self-Supervised Learning Be Boosted by Downstream Data? A Theoretical Analysis