Learning with Pseudo-Ensembles

Abstract

We formalize the notion of a pseudo-ensemble, a (possibly infinite) collection of child models spawned from a parent model by perturbing it according to some noise process. E.g., dropout (Hinton et. al, 2012) in a deep neural network trains a pseudo-ensemble of child subnetworks generated by randomly masking nodes in the parent network. We present a novel regularizer based on making the behavior of a pseudo-ensemble robust with respect to the noise process generating it. In the fully-supervised setting, our regularizer matches the performance of dropout. But, unlike dropout, our regularizer naturally extends to the semi-supervised setting, where it produces state-of-the-art results. We provide a case study in which we transform the Recursive Neural Tensor Network of (Socher et. al, 2013) into a pseudo-ensemble, which significantly improves its performance on a real-world sentiment analysis benchmark.

To appear in Advances in Neural Information Processing Systems 27 (NIPS 2014), Advances in Neural Information Processing Systems 27, Dec. 2014

References in corpus (8)

• Improving neural networks by preventing co-adaptation of feature detectors
• Distributed Representations of Sentences and Documents
• On the difficulty of training Recurrent Neural Networks
• Deep Generative Stochastic Networks Trainable by Backprop
• Robustness and Regularization of Support Vector Machines
• Dropout Training as Adaptive Regularization
• Large-Scale Feature Learning With Spike-and-Slab Sparse Coding
• Learning Ordered Representations with Nested Dropout

Cited by in corpus (85)

• Bootstrap your own latent: A new approach to self-supervised Learning
• Gaussian Error Linear Units (GELUs)
• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
• Unsupervised Data Augmentation for Consistency Training
• AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty
• FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Rethinking Pre-training and Self-training
• Distributional Smoothing with Virtual Adversarial Training
• Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks
• A Simple Semi-Supervised Learning Framework for Object Detection
• Learning Discrete Representations via Information Maximizing Self-Augmented Training
• Uncertainty-aware multi-view co-training for semi-supervised medical image segmentation and domain adaptation
• Self-training with Noisy Student improves ImageNet classification
• Rethinking the Value of Labels for Improving Class-Imbalanced Learning
• Realistic Evaluation of Deep Semi-Supervised Learning Algorithms
• Are Labels Required for Improving Adversarial Robustness?
• ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring
• Dropout as data augmentation
• A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation
• Dark Experience for General Continual Learning: a Strong, Simple Baseline
• Cycle Self-Training for Domain Adaptation
• Data-Efficient Reinforcement Learning with Self-Predictive Representations
• Consistency Regularization for Generative Adversarial Networks
• Data Distillation: Towards Omni-Supervised Learning
• Uncertainty-aware Self-training for Text Classification with Few Labels
• Noisy Student Training using Body Language Dataset Improves Facial Expression Recognition
• Class-Imbalanced Semi-Supervised Learning
• Adaptive Self-training for Few-shot Neural Sequence Labeling
• Improved Training of Physics-Informed Neural Networks with Model Ensembles
• Regularizing Class-wise Predictions via Self-knowledge Distillation
• CoDA: Contrast-enhanced and Diversity-promoting Data Augmentation for Natural Language Understanding
• Deep Co-Training for Semi-Supervised Image Recognition
• Improved Consistency Regularization for GANs
• Training Recurrent Neural Networks by Diffusion
• Smooth Neighbors on Teacher Graphs for Semi-supervised Learning
• SemiFed: Semi-supervised Federated Learning with Consistency and Pseudo-Labeling
• Twin Neural Network Regression
• ST++: Make Self-training Work Better for Semi-supervised Semantic Segmentation
• Dual Student: Breaking the Limits of the Teacher in Semi-supervised Learning
• Semi-Supervised Sequence Modeling with Cross-View Training
• On the Inductive Bias of Dropout
• Entanglement Verification with Deep Semi-supervised Machine Learning
• On Data-Augmentation and Consistency-Based Semi-Supervised Learning
• Test time Adaptation through Perturbation Robustness
• Exploring Self-Supervised Regularization for Supervised and Semi-Supervised Learning
• Ego-Vehicle Action Recognition based on Semi-Supervised Contrastive Learning
• Deep Ensembles for Low-Data Transfer Learning
• Generalized Negative Correlation Learning for Deep Ensembling
• SST: Self-training with Self-adaptive Thresholding for Semi-supervised Learning
• Multi-Task Recurrent Convolutional Network with Correlation Loss for Surgical Video Analysis
• Antipodes of Label Differential Privacy: PATE and ALIBI
• VFunc: a Deep Generative Model for Functions
• Sinkhorn Label Allocation: Semi-Supervised Classification via Annealed Self-Training
• Teacher Supervises Students How to Learn From Partially Labeled Images for Facial Landmark Detection
• A Realistic Evaluation of Semi-Supervised Learning for Fine-Grained Classification
• An analysis of over-sampling labeled data in semi-supervised learning with FixMatch
• Countering Noisy Labels By Learning From Auxiliary Clean Labels
• WSSOD: A New Pipeline for Weakly- and Semi-Supervised Object Detection
• Infinite Variational Autoencoder for Semi-Supervised Learning
• Co-matching: Combating Noisy Labels by Augmentation Anchoring
• Post-Estimation Smoothing: A Simple Baseline for Learning with Side Information
• Where is the Bottleneck of Adversarial Learning with Unlabeled Data?
• Infomax Neural Joint Source-Channel Coding via Adversarial Bit Flip
• Unsupervised Domain Adaptation using Generative Models and Self-ensembling
• Pi-NAS: Improving Neural Architecture Search by Reducing Supernet Training Consistency Shift
• Ranking Neural Checkpoints
• Self-similarity Student for Partial Label Histopathology Image Segmentation
• Surprising properties of dropout in deep networks
• Non-technical Loss Detection with Statistical Profile Images Based on Semi-supervised Learning
• Certainty Driven Consistency Loss on Multi-Teacher Networks for Semi-Supervised Learning
• Efficient Estimation of Influence of a Training Instance
• Improving Object Detection with Selective Self-supervised Self-training
• Multimodal Knowledge Expansion
• Semi-Supervised Vision Transformers
• Matching Distributions via Optimal Transport for Semi-Supervised Learning
• Noised Consistency Training for Text Summarization
• Hybrid BYOL-ViT: Efficient approach to deal with small datasets
• Discriminative Clustering with Representation Learning with any Ratio of Labeled to Unlabeled Data
• Semi-Supervised Deep Learning Using Improved Unsupervised Discriminant Projection
• Learning Identity-Preserving Transformations on Data Manifolds
• Semi-supervised Learning with Contrastive Predicative Coding
• RCT: Random Consistency Training for Semi-supervised Sound Event Detection
• SSLayout360: Semi-Supervised Indoor Layout Estimation from 360-Degree Panorama
• Semi-Supervised Learning with Taxonomic Labels