Spectral Pruning: Compressing Deep Neural Networks via Spectral Analysis and its Generalization Error

Abstract

Compression techniques for deep neural network models are becoming very important for the efficient execution of high-performance deep learning systems on edge-computing devices. The concept of model compression is also important for analyzing the generalization error of deep learning, known as the compression-based error bound. However, there is still huge gap between a practically effective compression method and its rigorous background of statistical learning theory. To resolve this issue, we develop a new theoretical framework for model compression and propose a new pruning method called {\it spectral pruning} based on this framework. We define the ``degrees of freedom'' to quantify the intrinsic dimensionality of a model by using the eigenvalue distribution of the covariance matrix across the internal nodes and show that the compression ability is essentially controlled by this quantity. Moreover, we present a sharp generalization error bound of the compressed model and characterize the bias--variance tradeoff induced by the compression procedure. We apply our method to several datasets to justify our theoretical analyses and show the superiority of the the proposed method.

arXiv comment17 pages, 4 figures. Accepted in IJCAI-PRICAI 2020. Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, pages 2839--2846

References in corpus (16)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• Compressing Neural Networks with the Hashing Trick
• Learning Structured Sparsity in Deep Neural Networks
• Dropout Training as Adaptive Regularization
• Spectrally-normalized margin bounds for neural networks
• Concentration inequalities and asymptotic results for ratio type empirical processes
• Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approach
• Norm-Based Capacity Control in Neural Networks
• Dual Path Networks
• Risk Bounds for High-dimensional Ridge Function Combinations Including Neural Networks
• Approximation by Combinations of ReLU and Squared ReLU Ridge Functions with ${\ell }^1$ and ${\ell }^0$ Controls
• On the Depth of Deep Neural Networks: A Theoretical View
• Fast learning rate of deep learning via a kernel perspective

Cited by in corpus (4)

• Model Pruning Enables Localized and Efficient Federated Learning for Yield Forecasting and Data Sharing
• CHIP: CHannel Independence-based Pruning for Compact Neural Networks
• Compression based bound for non-compressed network: unified generalization error analysis of large compressible deep neural network
• Understanding the Effects of Pre-Training for Object Detectors via Eigenspectrum