MMA Training: Direct Input Space Margin Maximization through Adversarial Training

Abstract

We study adversarial robustness of neural networks from a margin maximization perspective, where margins are defined as the distances from inputs to a classifier's decision boundary. Our study shows that maximizing margins can be achieved by minimizing the adversarial loss on the decision boundary at the "shortest successful perturbation", demonstrating a close connection between adversarial losses and the margins. We propose Max-Margin Adversarial (MMA) training to directly maximize the margins to achieve adversarial robustness. Instead of adversarial training with a fixed $\epsilon$, MMA offers an improvement by enabling adaptive selection of the "correct" $\epsilon$ as the margin individually for each datapoint. In addition, we rigorously analyze adversarial training with the perspective of margin maximization, and provide an alternative interpretation for adversarial training, maximizing either a lower or an upper bound of the margins. Our experiments empirically confirm our theory and demonstrate MMA training's efficacy on the MNIST and CIFAR10 datasets w.r.t. ${\ell }_{\infty }$ and ${\ell }_2$ robustness. Code and models are available at https://github.com/BorealisAI/mma_training.

arXiv commentPublished at the Eighth International Conference on Learning Representations (ICLR 2020), https://openreview.net/forum?id=HkeryxBtPB

References in corpus (13)

• Wide Residual Networks
• Theoretically Principled Trade-off between Robustness and Accuracy
• Using Pre-Training Can Improve Model Robustness and Uncertainty
• Adversarial Risk and the Dangers of Evaluating Against Weak Attacks
• Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradients
• Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulation
• Learning with a Strong Adversary
• Robust Large Margin Deep Neural Networks
• Are Labels Required for Improving Adversarial Robustness?
• Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks
• advertorch v0.1: An Adversarial Robustness Toolbox based on PyTorch
• Low Frequency Adversarial Perturbation
• The Limitations of Adversarial Training and the Blind-Spot Attack

Cited by in corpus (43)

• Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks
• RobustBench: a standardized adversarial robustness benchmark
• A Review of Adversarial Attack and Defense for Classification Methods
• Recent Advances in Adversarial Training for Adversarial Robustness
• Geometry-aware Instance-reweighted Adversarial Training
• Exploring Architectural Ingredients of Adversarially Robust Deep Neural Networks
• Once-for-All Adversarial Training: In-Situ Tradeoff between Robustness and Accuracy for Free
• Reliable Adversarial Distillation with Unreliable Teachers
• SAT: Improving Adversarial Training via Curriculum-Based Loss Smoothing
• Attacking Adversarial Attacks as A Defense
• Bag of Tricks for Adversarial Training
• Adversarial Example Games
• Adversarial Robustness under Long-Tailed Distribution
• Fighting Gradients with Gradients: Dynamic Defenses against Adversarial Attacks
• Increasing-Margin Adversarial (IMA) Training to Improve Adversarial Robustness of Neural Networks
• Adversarial Robustness through the Lens of Convolutional Filters
• BulletTrain: Accelerating Robust Neural Network Training via Boundary Example Mining
• SOAR: Second-Order Adversarial Regularization
• Guided Interpolation for Adversarial Training
• Boosting the Certified Robustness of L-infinity Distance Nets
• Towards Evaluating the Robustness of Neural Networks Learned by Transduction
• LAFEAT: Piercing Through Adversarial Defenses with Latent Features
• Towards Certifying L-infinity Robustness using Neural Networks with L-inf-dist Neurons
• Adaptive Feature Alignment for Adversarial Training
• ROBY: Evaluating the Robustness of a Deep Model by its Decision Boundaries
• Adversarial Sample Enhanced Domain Adaptation: A Case Study on Predictive Modeling with Electronic Health Records
• Multi-stage Optimization based Adversarial Training
• Robust Sensible Adversarial Learning of Deep Neural Networks for Image Classification
• Meta-Learning the Search Distribution of Black-Box Random Search Based Adversarial Attacks
• Blind Adversarial Pruning: Balance Accuracy, Efficiency and Robustness
• Enhance Diffusion to Improve Robust Generalization
• Colored Noise Injection for Training Adversarially Robust Neural Networks
• Adversarial Attacks on ML Defense Models Competition
• Lagrangian Objective Function Leads to Improved Unforeseen Attack Generalization in Adversarial Training
• Enhance CNN Robustness Against Noises for Classification of 12-Lead ECG with Variable Length
• Dynamic Efficient Adversarial Training Guided by Gradient Magnitude
• Enhanced countering adversarial attacks via input denoising and feature restoring
• MadNet: Using a MAD Optimization for Defending Against Adversarial Attacks
• Online Adversarial Attacks
• Bridged Adversarial Training
• Adaptive perturbation adversarial training: based on reinforcement learning
• Calibrated Adversarial Training
• Consistent Non-Parametric Methods for Maximizing Robustness