Swapout: Learning an ensemble of deep architectures

Abstract

We describe Swapout, a new stochastic training method, that outperforms ResNets of identical network structure yielding impressive results on CIFAR-10 and CIFAR-100. Swapout samples from a rich set of architectures including dropout, stochastic depth and residual architectures as special cases. When viewed as a regularization method swapout not only inhibits co-adaptation of units in a layer, similar to dropout, but also across network layers. We conjecture that swapout achieves strong regularization by implicitly tying the parameters across layers. When viewed as an ensemble training method, it samples a much richer set of architectures than existing methods such as dropout or stochastic depth. We propose a parameterization that reveals connections to exiting architectures and suggests a much richer set of architectures to be explored. We show that our formulation suggests an efficient training method and validate our conclusions on CIFAR-10 and CIFAR-100 matching state of the art accuracy. Remarkably, our 32 layer wider model performs similar to a 1001 layer ResNet model.

arXiv commentSubmitted to NIPS 2016

References in corpus (8)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Deep Residual Learning for Image Recognition
• Going Deeper with Convolutions
• Training Very Deep Networks
• Stochastic Pooling for Regularization of Deep Convolutional Neural Networks
• Train faster, generalize better: Stability of stochastic gradient descent
• Deep Networks with Stochastic Depth
• Recurrent Convolutional Neural Networks for Scene Parsing

Cited by in corpus (33)

• Ensemble deep learning: A review
• Temporal Ensembling for Semi-Supervised Learning
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• Residual Networks of Residual Networks: Multilevel Residual Networks
• SMASH: One-Shot Model Architecture Search through HyperNetworks
• Recent Advances in Convolutional Neural Networks
• Convolutional Neural Fabrics
• Machine and Deep Learning for IoT Security and Privacy: Applications, Challenges, and Future Directions
• Interleaved Group Convolutions for Deep Neural Networks
• Robust Self-Ensembling Network for Hyperspectral Image Classification
• OSLNet: Deep Small-Sample Classification with an Orthogonal Softmax Layer
• Survey of Dropout Methods for Deep Neural Networks
• Deep Convolutional Neural Network Design Patterns
• Multi-Residual Networks: Improving the Speed and Accuracy of Residual Networks
• PolyNet: A Pursuit of Structural Diversity in Very Deep Networks
• Deep Convolutional Neural Networks with Merge-and-Run Mappings
• Deep Pyramidal Residual Networks
• Approximate Inference with Amortised MCMC
• MEAL: Multi-Model Ensemble via Adversarial Learning
• Dynamic Optimization of Neural Network Structures Using Probabilistic Modeling
• Recent Advances in the Applications of Convolutional Neural Networks to Medical Image Contour Detection
• Convolutional Residual Memory Networks
• Stochastic Downsampling for Cost-Adjustable Inference and Improved Regularization in Convolutional Networks
• Beyond Forward Shortcuts: Fully Convolutional Master-Slave Networks (MSNets) with Backward Skip Connections for Semantic Segmentation
• Drop-Activation: Implicit Parameter Reduction and Harmonic Regularization
• DelugeNets: Deep Networks with Efficient and Flexible Cross-layer Information Inflows
• Adversarial-Based Knowledge Distillation for Multi-Model Ensemble and Noisy Data Refinement
• Supervised Deep Sparse Coding Networks
• ResiliNet: Failure-Resilient Inference in Distributed Neural Networks
• Adaptive Regularization via Residual Smoothing in Deep Learning Optimization
• Stability Enhanced Privacy and Applications in Private Stochastic Gradient Descent
• Reconciling Feature-Reuse and Overfitting in DenseNet with Specialized Dropout