Adversarial Robustness Assessment: Why both $L_0$ and $L_{\infty }$ Attacks Are Necessary

Abstract

There exists a vast number of adversarial attacks and defences for machine learning algorithms of various types which makes assessing the robustness of algorithms a daunting task. To make matters worse, there is an intrinsic bias in these adversarial algorithms. Here, we organise the problems faced: a) Model Dependence, b) Insufficient Evaluation, c) False Adversarial Samples, and d) Perturbation Dependent Results). Based on this, we propose a model agnostic dual quality assessment method, together with the concept of robustness levels to tackle them. We validate the dual quality assessment on state-of-the-art neural networks (WideResNet, ResNet, AllConv, DenseNet, NIN, LeNet and CapsNet) as well as adversarial defences for image classification problem. We further show that current networks and defences are vulnerable at all levels of robustness. The proposed robustness assessment reveals that depending on the metric used (i.e., $L_0$ or $L_{\infty }$), the robustness may vary significantly. Hence, the duality should be taken into account for a correct evaluation. Moreover, a mathematical derivation, as well as a counter-example, suggest that $L_1$ and $L_2$ metrics alone are not sufficient to avoid spurious adversarial samples. Interestingly, the threshold attack of the proposed assessment is a novel $L_{\infty }$ black-box adversarial method which requires even less perturbation than the One-Pixel Attack (only $12\%$ of One-Pixel Attack's amount of perturbation) to achieve similar results. Code is available at http://bit.ly/DualQualityAssessment.

References in corpus (13)

• Striving for Simplicity: The All Convolutional Net
• Ensemble Adversarial Training: Attacks and Defenses
• Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks
• Wide Residual Networks
• DenseNet: Implementing Efficient ConvNet Descriptor Pyramids
• Characterizing Adversarial Subspaces Using Local Intrinsic Dimensionality
• Adversarial Risk and the Dangers of Evaluating Against Weak Attacks
• On the (Statistical) Detection of Adversarial Examples
• Distributional Smoothing with Virtual Adversarial Training
• Learning with a Strong Adversary
• A study of the effect of JPG compression on adversarial images
• Keeping the Bad Guys Out: Protecting and Vaccinating Deep Learning with JPEG Compression
• MagNet and "Efficient Defenses Against Adversarial Attacks" are Not Robust to Adversarial Examples

Cited by in corpus (4)

• Evolving Robust Neural Architectures to Defend from Adversarial Attacks
• A Systematic Evaluation of Adversarial Attacks against Speech Emotion Recognition Models
• Representation Quality Of Neural Networks Links To Adversarial Attacks and Defences
• Adversarial Attacks with Time-Scale Representations