Data-Dependent Coresets for Compressing Neural Networks with Applications to Generalization Bounds

Abstract

We present an efficient coresets-based neural network compression algorithm that sparsifies the parameters of a trained fully-connected neural network in a manner that provably approximates the network's output. Our approach is based on an importance sampling scheme that judiciously defines a sampling distribution over the neural network parameters, and as a result, retains parameters of high importance while discarding redundant ones. We leverage a novel, empirical notion of sensitivity and extend traditional coreset constructions to the application of compressing parameters. Our theoretical analysis establishes guarantees on the size and accuracy of the resulting compressed network and gives rise to generalization bounds that may provide new insights into the generalization properties of neural networks. We demonstrate the practical effectiveness of our algorithm on a variety of neural network configurations and real-world data sets.

arXiv commentFirst two authors contributed equally

References in corpus (15)

• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Compressing Deep Convolutional Networks using Vector Quantization
• Compressing Neural Networks with the Hashing Trick
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Convolutional neural networks with low-rank regularization
• A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• Probabilistic Tools for the Analysis of Randomized Optimization Heuristics
• Compression-aware Training of Deep Networks
• Coresets for Scalable Bayesian Logistic Regression
• Structured Transforms for Small-Footprint Deep Learning
• Binary embeddings with structured hashed projections
• A Note on Randomized Element-wise Matrix Sparsification