On Data-Augmentation and Consistency-Based Semi-Supervised Learning

Abstract

Recently proposed consistency-based Semi-Supervised Learning (SSL) methods such as the $\Pi$-model, temporal ensembling, the mean teacher, or the virtual adversarial training, have advanced the state of the art in several SSL tasks. These methods can typically reach performances that are comparable to their fully supervised counterparts while using only a fraction of labelled examples. Despite these methodological advances, the understanding of these methods is still relatively limited. In this text, we analyse (variations of) the $\Pi$-model in settings where analytically tractable results can be obtained. We establish links with Manifold Tangent Classifiers and demonstrate that the quality of the perturbations is key to obtaining reasonable SSL performances. Importantly, we propose a simple extension of the Hidden Manifold Model that naturally incorporates data-augmentation schemes and offers a framework for understanding and experimenting with SSL methods.

References in corpus (18)

• A Simple Framework for Contrastive Learning of Visual Representations
• mixup: Beyond Empirical Risk Minimization
• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Bootstrap your own latent: A new approach to self-supervised Learning
• The Effectiveness of Data Augmentation in Image Classification using Deep Learning
• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
• Unsupervised Data Augmentation for Consistency Training
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Temporal Ensembling for Semi-Supervised Learning
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Opening the Black Box of Deep Neural Networks via Information
• AutoAugment: Learning Augmentation Policies from Data
• Smart Augmentation - Learning an Optimal Data Augmentation Strategy
• Learning with Pseudo-Ensembles
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• Unsupervised Neural Machine Translation
• Generalisation error in learning with random features and the hidden manifold model
• Deep Learning and Hierarchal Generative Models

Cited by in corpus (4)

• PARSE: Pairwise Alignment of Representations in Semi-Supervised EEG Learning for Emotion Recognition
• OpenMatch: Open-set Consistency Regularization for Semi-supervised Learning with Outliers
• Multimodal Semi-Supervised Learning for 3D Objects
• An Empirical Study and Analysis on Open-Set Semi-Supervised Learning