Invariance-inducing regularization using worst-case transformations suffices to boost accuracy and spatial robustness

Abstract

This work provides theoretical and empirical evidence that invariance-inducing regularizers can increase predictive accuracy for worst-case spatial transformations (spatial robustness). Evaluated on these adversarially transformed examples, we demonstrate that adding regularization on top of standard or adversarial training reduces the relative error by 20% for CIFAR10 without increasing the computational cost. This outperforms handcrafted networks that were explicitly designed to be spatial-equivariant. Furthermore, we observe for SVHN, known to have inherent variance in orientation, that robust training also improves standard accuracy on the test set. We prove that this no-trade-off phenomenon holds for adversarial examples from transformation groups in the infinite data limit.

References in corpus (8)

• Unsupervised Data Augmentation for Consistency Training
• Provable defenses against adversarial examples via the convex outer adversarial polytope
• Group Equivariant Convolutional Networks
• Exploring the Landscape of Spatial Robustness
• Rotation equivariant vector field networks
• Adversarial Training Can Hurt Generalization
• Towards Practical Verification of Machine Learning: The Case of Computer Vision Systems
• Robustness of Rotation-Equivariant Networks to Adversarial Perturbations

Cited by in corpus (4)

• Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
• AdaMatch: A Unified Approach to Semi-Supervised Learning and Domain Adaptation
• SenSeI: Sensitive Set Invariance for Enforcing Individual Fairness
• Understanding (Non-)Robust Feature Disentanglement and the Relationship Between Low- and High-Dimensional Adversarial Attacks