Layer-wise Learning of Stochastic Neural Networks with Information Bottleneck
arXiv:1712.01272 · doi:10.3390/e21100976
Abstract
Information Bottleneck (IB) is a generalization of rate-distortion theory that naturally incorporates compression and relevance trade-offs for learning. Though the original IB has been extensively studied, there has not been much understanding of multiple bottlenecks which better fit in the context of neural networks. In this work, we propose Information Multi-Bottlenecks (IMBs) as an extension of IB to multiple bottlenecks which has a direct application to training neural networks by considering layers as multiple bottlenecks and weights as parameterized encoders and decoders. We show that the multiple optimality of IMB is not simultaneously achievable for stochastic encoders. We thus propose a simple compromised scheme of IMB which in turn generalizes maximum likelihood estimate (MLE) principle in the context of stochastic neural networks. We demonstrate the effectiveness of IMB on classification tasks and adversarial robustness in MNIST and CIFAR10.
published in Entropy journal
References in corpus (9)
- ADADELTA: An Adaptive Learning Rate Method
- A Survey of Model Compression and Acceleration for Deep Neural Networks
- On Mutual Information Maximization for Representation Learning
- Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle
- Compressing Neural Networks using the Variational Information Bottleneck
- Multivariate Information Bottleneck
- Stronger generalization bounds for deep nets via a compression approach
- Techniques for Learning Binary Stochastic Feedforward Neural Networks
- Bayesian Optimization with Unknown Search Space