Bridging the Gap Between Adversarial Robustness and Optimization Bias

Abstract

We demonstrate that the choice of optimizer, neural network architecture, and regularizer significantly affect the adversarial robustness of linear neural networks, providing guarantees without the need for adversarial training. To this end, we revisit a known result linking maximally robust classifiers and minimum norm solutions, and combine it with recent results on the implicit bias of optimizers. First, we show that, under certain conditions, it is possible to achieve both perfect standard accuracy and a certain degree of robustness, simply by training an overparametrized model using the implicit bias of the optimization. In that regime, there is a direct relationship between the type of the optimizer and the attack to which the model is robust. To the best of our knowledge, this work is the first to study the impact of optimization methods such as sign gradient descent and proximal methods on adversarial robustness. Second, we characterize the robustness of linear convolutional models, showing that they resist attacks subject to a constraint on the Fourier-${\ell }_{\infty }$ norm. To illustrate these findings we design a novel Fourier-${\ell }_{\infty }$ attack that finds adversarial examples with controllable frequencies. We evaluate Fourier-${\ell }_{\infty }$ robustness of adversarially-trained deep CIFAR-10 models from the standard RobustBench benchmark and visualize adversarial perturbations.

arXiv commentNew CIFAR-10 experiments and Fourier attack variations

References in corpus (36)

• Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
• Theoretically Principled Trade-off between Robustness and Accuracy
• Provable defenses against adversarial examples via the convex outer adversarial polytope
• Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks
• Adversarially Robust Generalization Requires More Data
• Robustness May Be at Odds with Accuracy
• Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradients
• On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Models
• Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulation
• Robust Large Margin Deep Neural Networks
• Adversarial Weight Perturbation Helps Robust Generalization
• Adversarial Robustness through Local Linearization
• Characterizing Implicit Bias in Terms of Optimization Geometry
• Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples
• On Adaptive Attacks to Adversarial Example Defenses
• A Convex Relaxation Barrier to Tight Robustness Verification of Neural Networks
• RobustBench: a standardized adversarial robustness benchmark
• Gradient descent aligns the layers of deep linear networks
• Gradient Descent Maximizes the Margin of Homogeneous Neural Networks
• Smooth Adversarial Training
• Low Frequency Adversarial Perturbation
• The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regime
• Adversarial Robustness Against the Union of Multiple Perturbation Models
• First-order Adversarial Vulnerability of Neural Networks and Input Dimension
• Batch Normalization is a Cause of Adversarial Vulnerability
• Understanding and Mitigating the Tradeoff Between Robustness and Accuracy
• Beyond Pixel Norm-Balls: Parametric Adversaries using an Analytically Differentiable Renderer
• Margins, Shrinkage, and Boosting
• An Effective Anti-Aliasing Approach for Residual Networks
• Towards an Adversarially Robust Normalization Approach
• Should Adversarial Attacks Use Pixel p-Norm?
• Adversarial Color Enhancement: Generating Unrestricted Adversarial Images by Optimizing a Color Filter
• A Unifying View on Implicit Bias in Training Linear Neural Networks
• Hold me tight! Influence of discriminative features on deep network boundaries
• Increasing-Margin Adversarial (IMA) Training to Improve Adversarial Robustness of Neural Networks
• SOAR: Second-Order Adversarial Regularization

Cited by in corpus (1)

• RobustBench: a standardized adversarial robustness benchmark