On Connections between Regularizations for Improving DNN Robustness

Abstract

This paper analyzes regularization terms proposed recently for improving the adversarial robustness of deep neural networks (DNNs), from a theoretical point of view. Specifically, we study possible connections between several effective methods, including input-gradient regularization, Jacobian regularization, curvature regularization, and a cross-Lipschitz functional. We investigate them on DNNs with general rectified linear activations, which constitute one of the most prevalent families of models for image classification and a host of other machine learning applications. We shed light on essential ingredients of these regularizations and re-interpret their functionality. Through the lens of our study, more principled and efficient regularizations can possibly be invented in the near future.

arXiv commentAccepted by TPAMI

References in corpus (10)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Models
• Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
• Technical Report on the CleverHans v2.1.0 Adversarial Examples Library
• Adversarial Machine Learning at Scale
• Robustness May Be at Odds with Accuracy
• Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradients
• Robust Large Margin Deep Neural Networks
• Parseval Networks: Improving Robustness to Adversarial Examples
• Structured Bayesian Pruning via Log-Normal Multiplicative Noise