Mutual Adversarial Training: Learning together is better than going alone

Abstract

Recent studies have shown that robustness to adversarial attacks can be transferred across networks. In other words, we can make a weak model more robust with the help of a strong teacher model. We ask if instead of learning from a static teacher, can models "learn together" and "teach each other" to achieve better robustness? In this paper, we study how interactions among models affect robustness via knowledge distillation. We propose mutual adversarial training (MAT), in which multiple models are trained together and share the knowledge of adversarial examples to achieve improved robustness. MAT allows robust models to explore a larger space of adversarial samples, and find more robust feature spaces and decision boundaries. Through extensive experiments on CIFAR-10 and CIFAR-100, we demonstrate that MAT can effectively improve model robustness and outperform state-of-the-art methods under white-box attacks, bringing $\sim$8% accuracy gain to vanilla adversarial training (AT) under PGD-100 attacks. In addition, we show that MAT can also mitigate the robustness trade-off among different perturbation types, bringing as much as 13.1% accuracy gain to AT baselines against the union of $l_{\infty }$, $l_2$ and $l_1$ attacks. These results show the superiority of the proposed method and demonstrate that collaborative learning is an effective strategy for designing robust models.

arXiv commentUnder submission

References in corpus (16)

• Distilling the Knowledge in a Neural Network
• Explaining and Harnessing Adversarial Examples
• Ensemble Adversarial Training: Attacks and Defenses
• The Space of Transferable Adversarial Examples
• Spatially Transformed Adversarial Examples
• Defensive Distillation is Not Robust to Adversarial Examples
• Adversarially Robust Distillation
• Foolbox: A Python toolbox to benchmark the robustness of machine learning models
• advertorch v0.1: An Adversarial Robustness Toolbox based on PyTorch
• Adversarial Training and Robustness for Multiple Perturbations
• Improving Adversarial Robustness of Ensembles with Diversity Training
• Functional Adversarial Attacks
• Testing Robustness Against Unforeseen Adversaries
• (De)Randomized Smoothing for Certifiable Defense against Patch Attacks
• DVERGE: Diversifying Vulnerabilities for Enhanced Robust Generation of Ensembles
• Training Ensembles to Detect Adversarial Examples

Cited by in corpus (3)

• Interpolated Joint Space Adversarial Training for Robust and Generalizable Defenses
• SecMLOps: A Comprehensive Framework for Integrating Security Throughout the MLOps Lifecycle
• Releasing Inequality Phenomenon in ${\ell }_{\infty }$-norm Adversarial Training via Input Gradient Distillation