Active Subspace of Neural Networks: Structural Analysis and Universal Attacks

Abstract

Active subspace is a model reduction method widely used in the uncertainty quantification community. In this paper, we propose analyzing the internal structure and vulnerability and deep neural networks using active subspace. Firstly, we employ the active subspace to measure the number of "active neurons" at each intermediate layer and reduce the number of neurons from several thousands to several dozens. This motivates us to change the network structure and to develop a new and more compact network, referred to as {ASNet}, that has significantly fewer model parameters. Secondly, we propose analyzing the vulnerability of a neural network using active subspace and finding an additive universal adversarial attack vector that can misclassify a dataset with a high probability. Our experiments on CIFAR-10 show that ASNet can achieve 23.98$\times$ parameter and 7.30$\times$ flops reduction. The universal active subspace attack vector can achieve around 20% higher attack ratio compared with the existing approach in all of our numerical experiments. The PyTorch codes for this paper are available online.

References in corpus (24)

• Adam: A Method for Stochastic Optimization
• Distilling the Knowledge in a Neural Network
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• FitNets: Hints for Thin Deep Nets
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Recent Trends in Deep Learning Based Natural Language Processing
• Rethinking the Value of Network Pruning
• Tensorizing Neural Networks
• Group Sparse Regularization for Deep Neural Networks
• Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
• A study of the effect of JPG compression on adversarial images
• Adversarial Transformation Networks: Learning to Generate Adversarial Examples
• Exploiting Active Subspaces to Quantify Uncertainty in the Numerical Simulation of the HyShot II Scramjet
• Ultimate tensorization: compressing convolutional and FC layers alike
• Towards moderate overparameterization: global convergence guarantees for training shallow neural networks
• Adversarial Examples - A Complete Characterisation of the Phenomenon
• Universal Adversarial Audio Perturbations
• Progressive DNN Compression: A Key to Achieve Ultra-High Weight Pruning and Quantization Rates using ADMM
• Bayesian Tensorized Neural Networks with Automatic Rank Selection
• Mildly Overparametrized Neural Nets can Memorize Training Data Efficiently
• Dynamic Sparse Graph for Efficient Deep Learning
• Gradient Descent-based D-optimal Design for the Least-Squares Polynomial Approximation