$S^2$-LBI: Stochastic Split Linearized Bregman Iterations for Parsimonious Deep Learning

Abstract

This paper proposes a novel Stochastic Split Linearized Bregman Iteration ($S^2$-LBI) algorithm to efficiently train the deep network. The $S^2$-LBI introduces an iterative regularization path with structural sparsity. Our $S^2$-LBI combines the computational efficiency of the LBI, and model selection consistency in learning the structural sparsity. The computed solution path intrinsically enables us to enlarge or simplify a network, which theoretically, is benefited from the dynamics property of our $S^2$-LBI algorithm. The experimental results validate our $S^2$-LBI on MNIST and CIFAR-10 dataset. For example, in MNIST, we can either boost a network with only 1.5K parameters (1 convolutional layer of 5 filters, and 1 FC layer), achieves 98.40\% recognition accuracy; or we simplify $82.5\%$ of parameters in LeNet-5 network, and still achieves the 98.47\% recognition accuracy. In addition, we also have the learning results on ImageNet, which will be added in the next version of our report.

technical report

References in corpus (9)

• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Neural Architecture Search with Reinforcement Learning
• Trained Ternary Quantization
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
• Learning Efficient Convolutional Networks through Network Slimming
• Learning the Number of Neurons in Deep Networks
• Memory Bounded Deep Convolutional Networks
• Sparse Recovery via Differential Inclusions