Layer-wise Learning of Stochastic Neural Networks with Information Bottleneck
arXiv:1712.01272 · doi:10.3390/e21100976
Abstract
Information Bottleneck (IB) is a generalization of rate-distortion theory that naturally incorporates compression and relevance trade-offs for learning. Though the original IB has been extensively studied, there has not been much understanding of multiple bottlenecks which better fit in the context of neural networks. In this work, we propose Information Multi-Bottlenecks (IMBs) as an extension of IB to multiple bottlenecks which has a direct application to training neural networks by considering layers as multiple bottlenecks and weights as parameterized encoders and decoders. We show that the multiple optimality of IMB is not simultaneously achievable for stochastic encoders. We thus propose a simple compromised scheme of IMB which in turn generalizes maximum likelihood estimate (MLE) principle in the context of stochastic neural networks. We demonstrate the effectiveness of IMB on classification tasks and adversarial robustness in MNIST and CIFAR10.
arXiv commentpublished in Entropy journal
References in corpus (14)
- ADADELTA: An Adaptive Learning Rate Method
- The information bottleneck method
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- A Survey of Model Compression and Acceleration for Deep Neural Networks
- Opening the Black Box of Deep Neural Networks via Information
- Deep Variational Information Bottleneck
- On Mutual Information Maximization for Representation Learning
- Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle
- Stochastic Neural Networks for Hierarchical Reinforcement Learning
- Compressing Neural Networks using the Variational Information Bottleneck
- Multivariate Information Bottleneck
- Stronger generalization bounds for deep nets via a compression approach
- Techniques for Learning Binary Stochastic Feedforward Neural Networks
- Bayesian Optimization with Unknown Search Space