Adversarial Training Versus Weight Decay

Abstract

Performance-critical machine learning models should be robust to input perturbations not seen during training. Adversarial training is a method for improving a model's robustness to some perturbations by including them in the training process, but this tends to exacerbate other vulnerabilities of the model. The adversarial training framework has the effect of translating the data with respect to the cost function, while weight decay has a scaling effect. Although weight decay could be considered a crude regularization technique, it appears superior to adversarial training as it remains stable over a broader range of regimes and reduces all generalization errors. Equipped with these abstractions, we provide key baseline results and methodology for characterizing robustness. The two approaches can be combined to yield one small model that demonstrates good robustness to several white-box attacks associated with different metrics.

References in corpus (16)

• Ensemble Adversarial Training: Attacks and Defenses
• Wide Residual Networks
• Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
• The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks
• Technical Report on the CleverHans v2.1.0 Adversarial Examples Library
• Adversarial Risk and the Dangers of Evaluating Against Weak Attacks
• Certifying Some Distributional Robustness with Principled Adversarial Training
• Measuring the tendency of CNNs to Learn Surface Statistical Regularities
• Understanding Measures of Uncertainty for Adversarial Example Detection
• Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks
• A Boundary Tilting Persepective on the Phenomenon of Adversarial Examples
• Attacking the Madry Defense Model with $L_1$-based Adversarial Examples
• Norm-Based Capacity Control in Neural Networks
• Query-Efficient Black-box Adversarial Examples (superceded)
• High Dimensional Spaces, Deep Learning and Adversarial Examples
• Predicting Adversarial Examples with High Confidence

Cited by in corpus (10)

• Measuring Robustness to Natural Distribution Shifts in Image Classification
• Data augmentation instead of explicit regularization
• Batch Normalization is a Cause of Adversarial Vulnerability
• Adversarially Robust Training through Structured Gradient Regularization
• Built-in Vulnerabilities to Imperceptible Adversarial Perturbations
• Why ResNet Works? Residuals Generalize
• Data augmentation and image understanding
• How do SGD hyperparameters in natural training affect adversarial robustness?
• Robust Deep Neural Networks Inspired by Fuzzy Logic
• Adversarial Examples as an Input-Fault Tolerance Problem