Drill the Cork of Information Bottleneck by Inputting the Most Important Data
arXiv:2105.07181
Abstract
Deep learning has become the most powerful machine learning tool in the last decade. However, how to efficiently train deep neural networks remains to be thoroughly solved. The widely used minibatch stochastic gradient descent (SGD) still needs to be accelerated. As a promising tool to better understand the learning dynamic of minibatch SGD, the information bottleneck (IB) theory claims that the optimization process consists of an initial fitting phase and the following compression phase. Based on this principle, we further study typicality sampling, an efficient data selection method, and propose a new explanation of how it helps accelerate the training process of the deep networks. We show that the fitting phase depicted in the IB theory will be boosted with a high signal-to-noise ratio of gradient approximation if the typicality sampling is appropriately adopted. Furthermore, this finding also implies that the prior information of the training set is critical to the optimization process and the better use of the most important data can help the information flow through the bottleneck faster. Both theoretical analysis and experimental results on synthetic and real-world datasets demonstrate our conclusions.
arXiv comment11 pages, to be published in IEEE Transactions on Neural Networks and Learning Systems
References in corpus (15)
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- ADADELTA: An Adaptive Learning Rate Method
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- The information bottleneck method
- Understanding deep learning requires rethinking generalization
- Opening the Black Box of Deep Neural Networks via Information
- A Stochastic Gradient Method with an Exponential Convergence Rate for Finite Training Sets
- Deep Variational Information Bottleneck
- Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
- Variance Reduction in SGD by Distributed Importance Sampling
- Accelerating Minibatch Stochastic Gradient Descent using Stratified Sampling
- Big Batch SGD: Automated Inference using Adaptive Batch Sizes
- Flow of Information in Feed-Forward Deep Neural Networks
- Convergence diagnostics for stochastic gradient descent with constant step size
- Information Bottleneck Theory on Convolutional Neural Networks