Improving Adversarial Robustness for Free with Snapshot Ensemble

Abstract

Adversarial training, as one of the few certified defenses against adversarial attacks, can be quite complicated and time-consuming, while the results might not be robust enough. To address the issue of lack of robustness, ensemble methods were proposed, aiming to get the final output by weighting the selected results from repeatedly trained processes. It is proved to be very useful in achieving robust and accurate results, but the computational and memory costs are even higher. Snapshot ensemble, a new ensemble method that combines several local minima in a single training process to make the final prediction, was proposed recently, which reduces the time spent on training multiple networks and the memory to store the results. Based on the snapshot ensemble, we present a new method that is easier to implement: unlike original snapshot ensemble that seeks for local minima, our snapshot ensemble focuses on the last few iterations of a training and stores the sets of parameters from them. Our algorithm is much simpler but the results are no less accurate than the original ones: based on different hyperparameters and datasets, our snapshot ensemble has shown a 5% to 30% increase in accuracy when compared to the traditional adversarial training.

References in corpus (19)

• An overview of gradient descent optimization algorithms
• On the difficulty of training Recurrent Neural Networks
• Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Towards Deep Learning Models Resistant to Adversarial Attacks
• Provable defenses against adversarial examples via the convex outer adversarial polytope
• Certified Adversarial Robustness via Randomized Smoothing
• Characterizing Adversarial Subspaces Using Local Intrinsic Dimensionality
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• Variational Dropout and the Local Reparameterization Trick
• Learning with a Strong Adversary
• Nesterov Accelerated Gradient and Scale Invariance for Adversarial Attacks
• Averaging Weights Leads to Wider Optima and Better Generalization
• DeepFool: a simple and accurate method to fool deep neural networks
• Rethinking Softmax Cross-Entropy Loss for Adversarial Robustness
• A Fourier Perspective on Model Robustness in Computer Vision
• Adversarial Examples in Modern Machine Learning: A Review
• Robust Learning with Jacobian Regularization
• Differentially Private Bayesian Neural Networks on Accuracy, Privacy and Reliability