A Low Effort Approach to Structured CNN Design Using PCA

Abstract

Deep learning models hold state of the art performance in many fields, yet their design is still based on heuristics or grid search methods that often result in overparametrized networks. This work proposes a method to analyze a trained network and deduce an optimized, compressed architecture that preserves accuracy while keeping computational costs tractable. Model compression is an active field of research that targets the problem of realizing deep learning models in hardware. However, most pruning methodologies tend to be experimental, requiring large compute and time intensive iterations of retraining the entire network. We introduce structure into model design by proposing a single shot analysis of a trained network that serves as a first order, low effort approach to dimensionality reduction, by using PCA (Principal Component Analysis). The proposed method simultaneously analyzes the activations of each layer and considers the dimensionality of the space described by the filters generating these activations. It optimizes the architecture in terms of number of layers, and number of filters per layer without any iterative retraining procedures, making it a viable, low effort technique to design efficient networks. We demonstrate the proposed methodology on AlexNet and VGG style networks on the CIFAR-10, CIFAR-100 and ImageNet datasets, and successfully achieve an optimized architecture with a reduction of up to 3.8X and 9X in the number of operations and parameters respectively, while trading off less than 1% accuracy. We also apply the method to MobileNet, and achieve 1.7X and 3.9X reduction in the number of operations and parameters respectively, while improving accuracy by almost one percentage point.

arXiv commentTo be Published in IEEE Access, Volume 8, 2020

References in corpus (17)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Deep Residual Learning for Image Recognition
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• Predicting Parameters in Deep Learning
• Pruning Filters for Efficient ConvNets
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Practical recommendations for gradient-based training of deep architectures
• Learning Sparse Neural Networks through $L_0$ Regularization
• Compression-aware Training of Deep Networks
• PCA-Initialized Deep Neural Networks Applied To Document Image Analysis

Cited by in corpus (9)

• Constructing Energy-efficient Mixed-precision Neural Networks through Principal Component Analysis for Edge Intelligence
• Gradual Channel Pruning while Training using Feature Relevance Scores for Convolutional Neural Networks
• Gradient Projection Memory for Continual Learning
• Derivative-Informed Projected Neural Networks for High-Dimensional Parametric Maps Governed by PDEs
• Basis Scaling and Double Pruning for Efficient Inference in Network-Based Transfer Learning
• Principal Component Networks: Parameter Reduction Early in Training
• Activation Density driven Energy-Efficient Pruning in Training
• Complexity-aware Adaptive Training and Inference for Edge-Cloud Distributed AI Systems
• An Acceleration Method Based on Deep Learning and Multilinear Feature Space