Linking average- and worst-case perturbation robustness via class selectivity and dimensionality

Abstract

Representational sparsity is known to affect robustness to input perturbations in deep neural networks (DNNs), but less is known about how the semantic content of representations affects robustness. Class selectivity-the variability of a unit's responses across data classes or dimensions-is one way of quantifying the sparsity of semantic representations. Given recent evidence that class selectivity may not be necessary for, and in some cases can impair generalization, we investigate whether it also confers robustness (or vulnerability) to perturbations of input data. We found that networks regularized to have lower levels of class selectivity were more robust to average-case (naturalistic) perturbations, while networks with higher class selectivity are more vulnerable. In contrast, class selectivity increases robustness to multiple types of worst-case (i.e. white box adversarial) perturbations, suggesting that while decreasing class selectivity is helpful for average-case perturbations, it is harmful for worst-case perturbations. To explain this difference, we studied the dimensionality of the networks' representations: we found that the dimensionality of early-layer representations is inversely proportional to a network's class selectivity, and that adversarial samples cause a larger increase in early-layer dimensionality than corrupted samples. Furthermore, the input-unit gradient is more variable across samples and units in high-selectivity networks compared to low-selectivity networks. These results lead to the conclusion that units participate more consistently in low-selectivity regimes compared to high-selectivity regimes, effectively creating a larger attack surface and hence vulnerability to worst-case perturbations.

arXiv commentarXiv admin note: text overlap with arXiv:2007.04440

References in corpus (29)

• Explaining and Harnessing Adversarial Examples
• Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
• Visualizing and Understanding Recurrent Networks
• Characterizing Adversarial Subspaces Using Local Intrinsic Dimensionality
• Understanding the Role of Individual Units in a Deep Neural Network
• Estimating the intrinsic dimension of datasets by a minimal neighborhood information
• Learning to Generate Reviews and Discovering Sentiment
• Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods
• Sensitivity and Generalization in Neural Networks: an Empirical Study
• Learning with a Strong Adversary
• Robust Large Margin Deep Neural Networks
• On the importance of single directions for generalization
• Stochastic Activation Pruning for Robust Adversarial Defense
• Ablation Studies in Artificial Neural Networks
• Motivating the Rules of the Game for Adversarial Example Research
• Examining the Impact of Blur on Recognition by Convolutional Networks
• Adversarial Examples Are a Natural Consequence of Test Error in Noise
• Benchmarking Neural Network Robustness to Common Corruptions and Surface Variations
• Biologically inspired protection of deep networks from adversarial attacks
• Improving the Robustness of Deep Neural Networks via Stability Training
• Revisiting the Importance of Individual Units in CNNs via Ablation
• How Important Is a Neuron?
• Understanding trained CNNs by indexing neuron selectivity
• Adversarial Examples - A Complete Characterisation of the Phenomenon
• Robust Learning with Jacobian Regularization
• On the Limitation of Local Intrinsic Dimensionality for Characterizing the Subspaces of Adversarial Examples
• Selectivity considered harmful: evaluating the causal impact of class selectivity in DNNs
• Ablation of a Robot's Brain: Neural Networks Under a Knife
• On the Effect of Low-Rank Weights on Adversarial Robustness of Neural Networks