Semi-Supervised Learning with Ladder Networks

Abstract

We combine supervised learning with unsupervised learning in deep neural networks. The proposed model is trained to simultaneously minimize the sum of supervised and unsupervised cost functions by backpropagation, avoiding the need for layer-wise pre-training. Our work builds on the Ladder network proposed by Valpola (2015), which we extend by combining the model with supervision. We show that the resulting model reaches state-of-the-art performance in semi-supervised MNIST and CIFAR-10 classification, in addition to permutation-invariant MNIST classification with all labels.

Revised denoising function, updated results, fixed typos

References in corpus (11)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Striving for Simplicity: The All Convolutional Net
• Semi-Supervised Learning with Deep Generative Models
• Theano: new features and speed improvements
• Stacked What-Where Auto-encoders
• Blocks and Fuel: Frameworks for deep learning
• How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation
• Large-Scale Feature Learning With Spike-and-Slab Sparse Coding
• Techniques for Learning Binary Stochastic Feedforward Neural Networks
• Denoising autoencoder with modulated lateral connections learns invariant representations of natural images
• Lateral Connections in Denoising Autoencoders Support Supervised Learning

Cited by in corpus (198)

• Bootstrap your own latent: A new approach to self-supervised Learning
• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
• Unsupervised Data Augmentation for Consistency Training
• Recent advances and clinical applications of deep learning in medical image analysis
• Active Learning for Convolutional Neural Networks: A Core-Set Approach
• MixMatch: A Holistic Approach to Semi-Supervised Learning
• Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning
• Semi-Supervised Learning with Generative Adversarial Networks
• Stacked Hourglass Networks for Human Pose Estimation
• FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling
• Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning
• Rethinking Pre-training and Self-training
• Variational Autoencoder for Deep Learning of Images, Labels and Captions
• InfoVAE: Information Maximizing Variational Autoencoders
• Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey
• Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks
• A Simple Semi-Supervised Learning Framework for Object Detection
• Adversarial Autoencoders
• BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning
• Uncertainty-aware multi-view co-training for semi-supervised medical image segmentation and domain adaptation
• Ladder Variational Autoencoders
• An Overview of Deep Semi-Supervised Learning
• Self-training with Noisy Student improves ImageNet classification
• SPICE: Semantic Pseudo-labeling for Image Clustering
• Semi-Supervised Medical Image Segmentation via Learning Consistency under Transformations
• Regularization for Deep Learning: A Taxonomy
• Unsupervised Learning via Meta-Learning
• AutoNovel: Automatically Discovering and Learning Novel Visual Categories
• Revisiting Self-Training for Neural Sequence Generation
• Semi-Supervised Speech Emotion Recognition with Ladder Networks
• BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling
• Review: Deep Learning in Electron Microscopy
• Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters
• Adversarially Robust Generalization Just Requires More Unlabeled Data
• Self-paced and self-consistent co-training for semi-supervised image segmentation
• Active One-shot Learning
• Deconstructing the Ladder Network Architecture
• On denoising autoencoders trained to minimise binary cross-entropy
• Cycle Self-Training for Domain Adaptation
• Semi-supervised Learning with Deep Generative Models for Asset Failure Prediction
• Semi-supervised Feature Learning For Improving Writer Identification
• Semi-supervised Deep Kernel Learning: Regression with Unlabeled Data by Minimizing Predictive Variance
• A critical look at the current train/test split in machine learning
• Uncertainty-aware Self-training for Text Classification with Few Labels
• Noisy Student Training using Body Language Dataset Improves Facial Expression Recognition
• Conditional Gaussian Distribution Learning for Open Set Recognition
• Stacked Capsule Autoencoders
• Variational Information Distillation for Knowledge Transfer
• Automatically Discovering and Learning New Visual Categories with Ranking Statistics
• CReST: A Class-Rebalancing Self-Training Framework for Imbalanced Semi-Supervised Learning
• Generalizing Skills with Semi-Supervised Reinforcement Learning
• Adaptive Self-training for Few-shot Neural Sequence Labeling
• Semi-Supervised Learning of Bearing Anomaly Detection via Deep Variational Autoencoders
• Semi-Supervised QA with Generative Domain-Adaptive Nets
• Deep Semi-Supervised Anomaly Detection
• Weakly Supervised Semantic Point Cloud Segmentation:Towards 10X Fewer Labels
• CoDA: Contrast-enhanced and Diversity-promoting Data Augmentation for Natural Language Understanding
• Instant-Teaching: An End-to-End Semi-Supervised Object Detection Framework
• Image Deformation Meta-Networks for One-Shot Learning
• Regularizing Deep Networks with Semantic Data Augmentation
• Learning Kernel for Conditional Moment-Matching Discrepancy-based Image Classification
• Robust Federated Learning Through Representation Matching and Adaptive Hyper-parameters
• Hebbian Semi-Supervised Learning in a Sample Efficiency Setting
• ST++: Make Self-training Work Better for Semi-supervised Semantic Segmentation
• A Unified Survey on Anomaly, Novelty, Open-Set, and Out-of-Distribution Detection: Solutions and Future Challenges
• Dual Student: Breaking the Limits of the Teacher in Semi-supervised Learning
• R-SQAIR: Relational Sequential Attend, Infer, Repeat
• Improving Back-Propagation by Adding an Adversarial Gradient
• Image-to-Image Translation: Methods and Applications
• Fast Task Inference with Variational Intrinsic Successor Features
• A Survey on Self-supervised Pre-training for Sequential Transfer Learning in Neural Networks
• Unsupervised Representation Learning with Future Observation Prediction for Speech Emotion Recognition
• Inf-Net: Automatic COVID-19 Lung Infection Segmentation from CT Images
• When Semi-Supervised Learning Meets Transfer Learning: Training Strategies, Models and Datasets
• Semi-Supervised Learning by Label Gradient Alignment
• Semi-supervised learning based on generative adversarial network: a comparison between good GAN and bad GAN approach
• Robust Learning Under Label Noise With Iterative Noise-Filtering
• Improving Semi-supervised Federated Learning by Reducing the Gradient Diversity of Models
• Guided Collaborative Training for Pixel-wise Semi-Supervised Learning
• A Probabilistic Framework for Deep Learning
• EnAET: A Self-Trained framework for Semi-Supervised and Supervised Learning with Ensemble Transformations
• Supervised and Semi-Supervised Deep Neural Networks for CSI-Based Authentication
• Neural Simpletrons - Minimalistic Directed Generative Networks for Learning with Few Labels
• Learning to Generate with Memory
• FocalMix: Semi-Supervised Learning for 3D Medical Image Detection
• Multi-Task Attention-Based Semi-Supervised Learning for Medical Image Segmentation
• Rethinking Assumptions in Deep Anomaly Detection
• Monocular 3D Detection with Geometric Constraints Embedding and Semi-supervised Training
• Energy Models for Better Pseudo-Labels: Improving Semi-Supervised Classification with the 1-Laplacian Graph Energy
• Large Margin Deep Networks for Classification
• FedCon: A Contrastive Framework for Federated Semi-Supervised Learning
• Deep Co-Training for Semi-Supervised Image Segmentation
• SESS: Self-Ensembling Semi-Supervised 3D Object Detection
• Test time Adaptation through Perturbation Robustness
• Analyzing Differentiable Fuzzy Logic Operators
• Temporal Difference Variational Auto-Encoder
• Deep Metric Transfer for Label Propagation with Limited Annotated Data
• RealMix: Towards Realistic Semi-Supervised Deep Learning Algorithms
• Semi-Supervised Generation with Cluster-aware Generative Models
• FedSiam: Towards Adaptive Federated Semi-Supervised Learning
• Semi-Supervised Learning using Siamese Networks
• Federated Semi-Supervised Learning with Class Distribution Mismatch
• Triple Generative Adversarial Networks
• Convolutional Networks with Dense Connectivity
• Universal Weakly Supervised Segmentation by Pixel-to-Segment Contrastive Learning
• Learning Domain Invariant Representations by Joint Wasserstein Distance Minimization
• The Rich Get Richer: Disparate Impact of Semi-Supervised Learning
• Progressive Cross-camera Soft-label Learning for Semi-supervised Person Re-identification
• ESAD: End-to-end Deep Semi-supervised Anomaly Detection
• Weakly Supervised Clustering by Exploiting Unique Class Count
• ALOHA: Auxiliary Loss Optimization for Hypothesis Augmentation
• Classification-Reconstruction Learning for Open-Set Recognition
• Deep Semi-Supervised Learning for Time Series Classification
• Generative Models For Deep Learning with Very Scarce Data
• A Bayesian Perspective of Convolutional Neural Networks through a Deconvolutional Generative Model
• DADA: Deep Adversarial Data Augmentation for Extremely Low Data Regime Classification
• A Realistic Evaluation of Semi-Supervised Learning for Fine-Grained Classification
• Self-supervised Semi-supervised Learning for Data Labeling and Quality Evaluation
• Deep Cross Residual Learning for Multitask Visual Recognition
• Semi-Supervised Phoneme Recognition with Recurrent Ladder Networks
• Adversarial Self-Supervised Learning for Semi-Supervised 3D Action Recognition
• Gaussian Gated Linear Networks
• Deep Low-Density Separation for Semi-Supervised Classification
• WSSOD: A New Pipeline for Weakly- and Semi-Supervised Object Detection
• Disentangling to Cluster: Gaussian Mixture Variational Ladder Autoencoders
• On Deep Representation Learning from Noisy Web Images
• A Biologically Inspired Visual Working Memory for Deep Networks
• Target-Embedding Autoencoders for Supervised Representation Learning
• Semi-Supervised Learning with the Deep Rendering Mixture Model
• Frequency Domain Transformer Networks for Video Prediction
• Auto-clustering Output Layer: Automatic Learning of Latent Annotations in Neural Networks
• A Self-supervised Approach to Hierarchical Forecasting with Applications to Groupwise Synthetic Controls
• One-bit Supervision for Image Classification
• Semi-supervised Learning using Adversarial Training with Good and Bad Samples
• Scaling Object Detection by Transferring Classification Weights
• A*HAR: A New Benchmark towards Semi-supervised learning for Class-imbalanced Human Activity Recognition
• Temporal Self-Ensembling Teacher for Semi-Supervised Object Detection
• Improving Embedding Extraction for Speaker Verification with Ladder Network
• Reconstructing the Noise Manifold for Image Denoising
• Mixup-breakdown: a consistency training method for improving generalization of speech separation models
• Label Prediction Framework for Semi-Supervised Cross-Modal Retrieval
• A Probabilistic Model for Discriminative and Neuro-Symbolic Semi-Supervised Learning
• Training Object Detectors from Few Weakly-Labeled and Many Unlabeled Images
• Optimally Combining Classifiers for Semi-Supervised Learning
• Renal Cell Carcinoma Detection and Subtyping with Minimal Point-Based Annotation in Whole-Slide Images
• Local Clustering with Mean Teacher for Semi-supervised Learning
• Ranking Neural Checkpoints
• Understanding and Improving Virtual Adversarial Training
• Distill-2MD-MTL: Data Distillation based on Multi-Dataset Multi-Domain Multi-Task Frame Work to Solve Face Related Tasksks, Multi Task Learning, Semi-Supervised Learning
• HybridNet: Classification and Reconstruction Cooperation for Semi-Supervised Learning
• Towards Robust Pattern Recognition: A Review
• PIMNet: A Parallel, Iterative and Mimicking Network for Scene Text Recognition
• AlphaMatch: Improving Consistency for Semi-supervised Learning with Alpha-divergence
• Matching Distributions via Optimal Transport for Semi-Supervised Learning
• Knowledge-Guided Object Discovery with Acquired Deep Impressions
• Semi-supervised Learning for COVID-19 Image Classification via ResNet
• A generic ensemble based deep convolutional neural network for semi-supervised medical image segmentation
• Reliable Label Bootstrapping for Semi-Supervised Learning
• Learning from Demonstration with Weakly Supervised Disentanglement
• Active Crowd Counting with Limited Supervision
• Improving Object Detection with Selective Self-supervised Self-training
• Learning with Multiplicative Perturbations
• Matching Embeddings for Domain Adaptation
• Learning Temporal Action Proposals With Fewer Labels
• Certainty Driven Consistency Loss on Multi-Teacher Networks for Semi-Supervised Learning
• Computational Graph Approach for Detection of Composite Human Activities
• Improved Generalization of Heading Direction Estimation for Aerial Filming Using Semi-supervised Regression
• ROI Regularization for Semi-supervised and Supervised Learning
• Single-Solution Hypervolume Maximization and its use for Improving Generalization of Neural Networks
• Spatial Mixture Models with Learnable Deep Priors for Perceptual Grouping
• Superpixel Sampling Networks
• Semi-Supervised Deep Learning Using Improved Unsupervised Discriminant Projection
• Learning Dual Retrieval Module for Semi-supervised Relation Extraction
• Semi-Supervised Self-Growing Generative Adversarial Networks for Image Recognition
• Smart-Inspect: Micro Scale Localization and Classification of Smartphone Glass Defects for Industrial Automation
• Snowball: Iterative Model Evolution and Confident Sample Discovery for Semi-Supervised Learning on Very Small Labeled Datasets
• Investigating the Effect of Intraclass Variability in Temporal Ensembling
• FLiText: A Faster and Lighter Semi-Supervised Text Classification with Convolution Networks
• Image Disguise based on Generative Model
• Image Embedded Segmentation: Uniting Supervised and Unsupervised Objectives for Segmenting Histopathological Images
• Improving Distantly-supervised Entity Typing with Compact Latent Space Clustering
• Adversarial Ladder Networks
• Variational Predictive Routing with Nested Subjective Timescales
• Semi-supervised Learning with Contrastive Predicative Coding
• CycleCluster: Modernising Clustering Regularisation for Deep Semi-Supervised Classification
• Scale Normalization
• Boosting Unconstrained Face Recognition with Auxiliary Unlabeled Data
• Iterative Self-Learning: Semi-Supervised Improvement to Dataset Volumes and Model Accuracy
• Semi-Supervised Learning with Taxonomic Labels
• Semi-Supervised Learning with Meta-Gradient
• Efficient data augmentation using graph imputation neural networks
• An empirical study of domain-agnostic semi-supervised learning via energy-based models: joint-training and pre-training
• Open-World Class Discovery with Kernel Networks
• SSLayout360: Semi-Supervised Indoor Layout Estimation from 360-Degree Panorama
• Guided Point Contrastive Learning for Semi-supervised Point Cloud Semantic Segmentation
• Domain Constraint Approximation based Semi Supervision
• Deep neural network based i-vector mapping for speaker verification using short utterances
• Compressive Sensing for cut improvement and local clustering