Interpolation Consistency Training for Semi-Supervised Learning

Abstract

We introduce Interpolation Consistency Training (ICT), a simple and computation efficient algorithm for training Deep Neural Networks in the semi-supervised learning paradigm. ICT encourages the prediction at an interpolation of unlabeled points to be consistent with the interpolation of the predictions at those points. In classification problems, ICT moves the decision boundary to low-density regions of the data distribution. Our experiments show that ICT achieves state-of-the-art performance when applied to standard neural network architectures on the CIFAR-10 and SVHN benchmark datasets. Our theoretical analysis shows that ICT corresponds to a certain type of data-adaptive regularization with unlabeled points which reduces overfitting to labeled points under high confidence values.

This is the latest version, which is published in the Journal, "Neural Networks", in 2022. All the previous results are unchanged. Keyword: Deep Learning, Semi-supervised Learning, Mixup

References in corpus (11)

• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Wide Residual Networks
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Temporal Ensembling for Semi-Supervised Learning
• Adversarially Learned Inference
• Manifold Mixup: Better Representations by Interpolating Hidden States
• There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average
• Learning Theory and Algorithms for Revenue Optimization in Second-Price Auctions with Reserve
• Adversarial Robustness May Be at Odds With Simplicity
• Understanding and Improving Interpolation in Autoencoders via an Adversarial Regularizer
• Manifold regularization with GANs for semi-supervised learning

Cited by in corpus (92)

• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
• Unsupervised Data Augmentation for Consistency Training
• A Survey on Deep Semi-supervised Learning
• MixMatch: A Holistic Approach to Semi-Supervised Learning
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Rethinking Pre-training and Self-training
• Hard Negative Mixing for Contrastive Learning
• Self-training with Noisy Student improves ImageNet classification
• A survey on Semi-, Self- and Unsupervised Learning for Image Classification
• Rethinking the Value of Labels for Improving Class-Imbalanced Learning
• ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring
• Semi-supervised semantic segmentation needs strong, varied perturbations
• InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization
• Selfie: Self-supervised Pretraining for Image Embedding
• PARSE: Pairwise Alignment of Representations in Semi-Supervised EEG Learning for Emotion Recognition
• Remix: Rebalanced Mixup
• MutexMatch: Semi-Supervised Learning with Mutex-Based Consistency Regularization
• Mixup Inference: Better Exploiting Mixup to Defend Adversarial Attacks
• ABC: Auxiliary Balanced Classifier for Class-imbalanced Semi-supervised Learning
• Class-Imbalanced Semi-Supervised Learning
• Exploring Feature Representation Learning for Semi-supervised Medical Image Segmentation
• A non-cooperative meta-modeling game for automated third-party calibrating, validating, and falsifying constitutive laws with parallelized adversarial attacks
• Semi-Supervised Building Footprint Generation with Feature and Output Consistency Training
• Interpolated Adversarial Training: Achieving Robust Neural Networks without Sacrificing Too Much Accuracy
• Efficient Semi-Supervised Gross Target Volume of Nasopharyngeal Carcinoma Segmentation via Uncertainty Rectified Pyramid Consistency
• Regularizing Deep Networks with Semantic Data Augmentation
• OpenMatch: Open-set Consistency Regularization for Semi-supervised Learning with Outliers
• How Does Mixup Help With Robustness and Generalization?
• Semi-Supervised Visual Representation Learning for Fashion Compatibility
• LaplaceNet: A Hybrid Graph-Energy Neural Network for Deep Semi-Supervised Classification
• Joint framework with deep feature distillation and adaptive focal loss for weakly supervised audio tagging and acoustic event detection
• EnAET: A Self-Trained framework for Semi-Supervised and Supervised Learning with Ensemble Transformations
• Cross-Referencing Self-Training Network for Sound Event Detection in Audio Mixtures
• Consistency-based Semi-supervised Active Learning: Towards Minimizing Labeling Cost
• SimPLE: Similar Pseudo Label Exploitation for Semi-Supervised Classification
• Energy Models for Better Pseudo-Labels: Improving Semi-Supervised Classification with the 1-Laplacian Graph Energy
• Interpolation-based semi-supervised learning for object detection
• Deep Partition Aggregation: Provable Defense against General Poisoning Attacks
• Robust Mutual Learning for Semi-supervised Semantic Segmentation
• Federated Semi-Supervised Learning for COVID Region Segmentation in Chest CT using Multi-National Data from China, Italy, Japan
• Neural Semi-supervised Learning for Text Classification Under Large-Scale Pretraining
• Exploring Self-Supervised Regularization for Supervised and Semi-Supervised Learning
• Dynamic Distillation Network for Cross-Domain Few-Shot Recognition with Unlabeled Data
• FeatMatch: Feature-Based Augmentation for Semi-Supervised Learning
• Achieving Generalizable Robustness of Deep Neural Networks by Stability Training
• Triple Generative Adversarial Networks
• Towards Domain-Agnostic Contrastive Learning
• KuroNet: Pre-Modern Japanese Kuzushiji Character Recognition with Deep Learning
• Adaptive Consistency Regularization for Semi-Supervised Transfer Learning
• Contrastive Semi-Supervised Learning for 2D Medical Image Segmentation
• XLST: Cross-lingual Self-training to Learn Multilingual Representation for Low Resource Speech Recognition
• A Three-Stage Self-Training Framework for Semi-Supervised Semantic Segmentation
• Self-supervised Semi-supervised Learning for Data Labeling and Quality Evaluation
• DINE: Domain Adaptation from Single and Multiple Black-box Predictors
• Learning from Label Proportions with Consistency Regularization
• Semi-Supervised Learning with Normalizing Flows
• Mask-based Data Augmentation for Semi-supervised Semantic Segmentation
• CMTS: Conditional Multiple Trajectory Synthesizer for Generating Safety-critical Driving Scenarios
• Density-Aware Graph for Deep Semi-Supervised Visual Recognition
• Negative sampling in semi-supervised learning
• Recent Deep Semi-supervised Learning Approaches and Related Works
• Robust Training of Social Media Image Classification Models for Rapid Disaster Response
• ConDA: Unsupervised Domain Adaptation for LiDAR Segmentation via Regularized Domain Concatenation
• Mixup Regularization for Region Proposal based Object Detectors
• Learning Invariant Representation with Consistency and Diversity for Semi-supervised Source Hypothesis Transfer
• The Mathematical Foundations of Manifold Learning
• Semi-Supervised Crowd Counting from Unlabeled Data
• Adversarial Transformations for Semi-Supervised Learning
• HODGEPODGE: Sound event detection based on ensemble of semi-supervised learning methods
• Mixup-breakdown: a consistency training method for improving generalization of speech separation models
• Target Consistency for Domain Adaptation: when Robustness meets Transferability
• Progressive Representative Labeling for Deep Semi-Supervised Learning
• SSMD: Semi-Supervised Medical Image Detection with Adaptive Consistency and Heterogeneous Perturbation
• Consistency Regularization with Generative Adversarial Networks for Semi-Supervised Learning
• Overcoming the curse of dimensionality with Laplacian regularization in semi-supervised learning
• Gradient-based Data Augmentation for Semi-Supervised Learning
• Active Crowd Counting with Limited Supervision
• Semixup: In- and Out-of-Manifold Regularization for Deep Semi-Supervised Knee Osteoarthritis Severity Grading from Plain Radiographs
• Enhancing Data-Free Adversarial Distillation with Activation Regularization and Virtual Interpolation
• CycleCluster: Modernising Clustering Regularisation for Deep Semi-Supervised Classification
• Epsilon Consistent Mixup: Structural Regularization with an Adaptive Consistency-Interpolation Tradeoff
• MetaMix: Improved Meta-Learning with Interpolation-based Consistency Regularization
• Integration of Unpaired Single-cell Chromatin Accessibility and Gene Expression Data via Adversarial Learning
• Negative Pseudo Labeling using Class Proportion for Semantic Segmentation in Pathology
• OBoW: Online Bag-of-Visual-Words Generation for Self-Supervised Learning
• RCT: Random Consistency Training for Semi-supervised Sound Event Detection
• End-To-End Graph-based Deep Semi-Supervised Learning
• Semi-supervised Learning of Fetal Anatomy from Ultrasound
• SketchTransfer: A Challenging New Task for Exploring Detail-Invariance and the Abstractions Learned by Deep Networks
• Hodge and Podge: Hybrid Supervised Sound Event Detection with Multi-Hot MixMatch and Composition Consistence Training
• Mapping Temporary Slums from Satellite Imagery using a Semi-Supervised Approach
• FROST: Faster and more Robust One-shot Semi-supervised Training