Learning with Pseudo-Ensembles

Abstract

We formalize the notion of a pseudo-ensemble, a (possibly infinite) collection of child models spawned from a parent model by perturbing it according to some noise process. E.g., dropout (Hinton et. al, 2012) in a deep neural network trains a pseudo-ensemble of child subnetworks generated by randomly masking nodes in the parent network. We present a novel regularizer based on making the behavior of a pseudo-ensemble robust with respect to the noise process generating it. In the fully-supervised setting, our regularizer matches the performance of dropout. But, unlike dropout, our regularizer naturally extends to the semi-supervised setting, where it produces state-of-the-art results. We provide a case study in which we transform the Recursive Neural Tensor Network of (Socher et. al, 2013) into a pseudo-ensemble, which significantly improves its performance on a real-world sentiment analysis benchmark.

To appear in Advances in Neural Information Processing Systems 27 (NIPS 2014), Advances in Neural Information Processing Systems 27, Dec. 2014

References in corpus (4)

• Improving neural networks by preventing co-adaptation of feature detectors
• Distributed Representations of Sentences and Documents
• On the difficulty of training Recurrent Neural Networks
• Large-Scale Feature Learning With Spike-and-Slab Sparse Coding

Cited by in corpus (29)

• Bootstrap your own latent: A new approach to self-supervised Learning
• AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty
• Uncertainty-aware multi-view co-training for semi-supervised medical image segmentation and domain adaptation
• Rethinking the Value of Labels for Improving Class-Imbalanced Learning
• ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring
• Consistency Regularization for Generative Adversarial Networks
• Data Distillation: Towards Omni-Supervised Learning
• Uncertainty-aware Self-training for Text Classification with Few Labels
• Class-Imbalanced Semi-Supervised Learning
• Regularizing Class-wise Predictions via Self-knowledge Distillation
• SemiFed: Semi-supervised Federated Learning with Consistency and Pseudo-Labeling
• On Data-Augmentation and Consistency-Based Semi-Supervised Learning
• Test time Adaptation through Perturbation Robustness
• Generalized Negative Correlation Learning for Deep Ensembling
• Antipodes of Label Differential Privacy: PATE and ALIBI
• Multi-Task Recurrent Convolutional Network with Correlation Loss for Surgical Video Analysis
• A Realistic Evaluation of Semi-Supervised Learning for Fine-Grained Classification
• An analysis of over-sampling labeled data in semi-supervised learning with FixMatch
• WSSOD: A New Pipeline for Weakly- and Semi-Supervised Object Detection
• Infinite Variational Autoencoder for Semi-Supervised Learning
• Where is the Bottleneck of Adversarial Learning with Unlabeled Data?
• Infomax Neural Joint Source-Channel Coding via Adversarial Bit Flip
• Co-matching: Combating Noisy Labels by Augmentation Anchoring
• Pi-NAS: Improving Neural Architecture Search by Reducing Supernet Training Consistency Shift
• Non-technical Loss Detection with Statistical Profile Images Based on Semi-supervised Learning
• Self-similarity Student for Partial Label Histopathology Image Segmentation
• Improving Object Detection with Selective Self-supervised Self-training
• Semi-Supervised Deep Learning Using Improved Unsupervised Discriminant Projection
• Semi-supervised Learning with Contrastive Predicative Coding