Inductive Bias of Gradient Descent based Adversarial Training on Separable Data

Abstract

Adversarial training is a principled approach for training robust neural networks. Despite of tremendous successes in practice, its theoretical properties still remain largely unexplored. In this paper, we provide new theoretical insights of gradient descent based adversarial training by studying its computational properties, specifically on its inductive bias. We take the binary classification task on linearly separable data as an illustrative example, where the loss asymptotically attains its infimum as the parameter diverges to infinity along certain directions. Specifically, we show that when the adversarial perturbation during training has bounded ${\ell }_2$-norm, the classifier learned by gradient descent based adversarial training converges in direction to the maximum ${\ell }_2$-norm margin classifier at the rate of $\tilde{\mathcal{O}}(1/\sqrt{T})$, significantly faster than the rate $\mathcal{O}(1/\log T)$ of training with clean data. In addition, when the adversarial perturbation during training has bounded ${\ell }_q$-norm for some $q\ge 1$, the resulting classifier converges in direction to a maximum mixed-norm margin classifier, which has a natural interpretation of robustness, as being the maximum ${\ell }_2$-norm margin classifier under worst-case ${\ell }_q$-norm perturbation to the data. Our findings provide theoretical backups for adversarial training that it indeed promotes robustness against adversarial perturbation.

References in corpus (12)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• The Loss Surfaces of Multilayer Networks
• Robustness and Regularization of Support Vector Machines
• Defensive Distillation is Not Robust to Adversarial Examples
• Characterizing Implicit Bias in Terms of Optimization Geometry
• Path-SGD: Path-Normalized Optimization in Deep Neural Networks
• Gradient descent aligns the layers of deep linear networks
• Feature Denoising for Improving Adversarial Robustness
• Norm-Based Capacity Control in Neural Networks
• Risk and parameter convergence of logistic regression
• Margins, Shrinkage, and Boosting
• Convergence and Margin of Adversarial Training on Separable Data

Cited by in corpus (4)

• Adversarial Examples Improve Image Recognition
• Understanding Generalization in Adversarial Training via the Bias-Variance Decomposition
• Improved OOD Generalization via Adversarial Training and Pre-training
• Adversarial Robustness of Supervised Sparse Coding