Intra-Ensemble in Neural Networks
arXiv:1904.04466
Abstract
Improving model performance is always the key problem in machine learning including deep learning. However, stand-alone neural networks always suffer from marginal effect when stacking more layers. At the same time, ensemble is an useful technique to further enhance model performance. Nevertheless, training several independent deep neural networks for ensemble costs multiple resources. If so, is it possible to utilize ensemble in only one neural network? In this work, we propose Intra-Ensemble, an end-to-end ensemble strategy with stochastic channel recombination operations to train several sub-networks simultaneously within one neural network. Additional parameter size is marginal since the majority of parameters are mutually shared. Meanwhile, stochastic channel recombination significantly increases the diversity of sub-networks, which finally enhances ensemble performance. Extensive experiments and ablation studies prove the applicability of intra-ensemble on various kinds of datasets and network architectures.
References in corpus (17)
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Neural Architecture Search with Reinforcement Learning
- Improved Regularization of Convolutional Neural Networks with Cutout
- MobileNetV2: Inverted Residuals and Linear Bottlenecks
- Wide Residual Networks
- Neural Architecture Search: A Survey
- DARTS: Differentiable Architecture Search
- Fine-Grained Visual Classification of Aircraft
- Efficient Neural Architecture Search via Parameter Sharing
- SMASH: One-Shot Model Architecture Search through HyperNetworks
- Shake-Shake regularization
- Slimmable Neural Networks
- Simple And Efficient Architecture Search for Convolutional Neural Networks
- Net2Net: Accelerating Learning via Knowledge Transfer
- FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search
- Incremental Training of Deep Convolutional Neural Networks
- IRLAS: Inverse Reinforcement Learning for Architecture Search