Task dependent Deep LDA pruning of neural networks
arXiv:1803.08134 · doi:10.1016/j.cviu.2020.103154
Abstract
With deep learning's success, a limited number of popular deep nets have been widely adopted for various vision tasks. However, this usually results in unnecessarily high complexities and possibly many features of low task utility. In this paper, we address this problem by introducing a task-dependent deep pruning framework based on Fisher's Linear Discriminant Analysis (LDA). The approach can be applied to convolutional, fully-connected, and module-based deep network structures, in all cases leveraging the high decorrelation of neuron motifs found in the pre-decision space and cross-layer deconv dependency. Moreover, we examine our approach's potential in network architecture search for specific tasks and analyze the influence of our pruning on model robustness to noises and adversarial attacks. Experimental results on datasets of generic objects (ImageNet, CIFAR100) as well as domain specific tasks (Adience, and LFWA) illustrate our framework's superior performance over state-of-the-art pruning approaches and fixed compact nets (e.g. SqueezeNet, MobileNet). The proposed method successfully maintains comparable accuracies even after discarding most parameters (98%-99% for VGG16, up to 82% for the already compact InceptionNet) and with significant FLOP reductions (83% for VGG16, up to 64% for InceptionNet). Through pruning, we can also derive smaller, but more accurate and more robust models suitable for the task.
References in corpus (26)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Distilling the Knowledge in a Neural Network
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Explaining and Harnessing Adversarial Examples
- SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
- Intriguing properties of neural networks
- Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
- Towards Deep Learning Models Resistant to Adversarial Attacks
- Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
- Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
- Delving into Transferable Adversarial Examples and Black-box Attacks
- Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
- Pruning Filters for Efficient ConvNets
- Speeding up Convolutional Neural Networks with Low Rank Expansions
- The Space of Transferable Adversarial Examples
- Bayesian Compression for Deep Learning
- Better Mixing via Deep Representations
- Discrimination-aware Channel Pruning for Deep Neural Networks
- Foolbox: A Python toolbox to benchmark the robustness of machine learning models
- EIE: Efficient Inference Engine on Compressed Deep Neural Network
- Training Skinny Deep Neural Networks with Iterative Hard Thresholding Methods
- Black-Box Adversarial Attack with Transferable Model-based Embedding
- Structured Pruning of Deep Convolutional Neural Networks
- Efficient Gender Classification Using a Deep LDA-Pruned Net
- Lookahead: A Far-Sighted Alternative of Magnitude-based Pruning