The coupling effect of Lipschitz regularization in deep neural networks
arXiv:1904.06253
Abstract
We investigate robustness of deep feed-forward neural networks when input data are subject to random uncertainties. More specifically, we consider regularization of the network by its Lipschitz constant and emphasize its role. We highlight the fact that this regularization is not only a way to control the magnitude of the weights but has also a coupling effect on the network weights accross the layers. We claim and show evidence on a dataset that this coupling effect brings a tradeoff between robustness and expressiveness of the network. This suggests that Lipschitz regularization should be carefully implemented so as to maintain coupling accross layers.
References in corpus (6)
- Understanding Adversarial Training: Increasing Local Stability of Neural Nets through Robust Optimization
- Robustness and Regularization of Support Vector Machines
- Spectral Norm Regularization for Improving the Generalizability of Deep Learning
- Lipschitz regularized Deep Neural Networks generalize and are adversarially robust
- Improved robustness to adversarial examples using Lipschitz regularization of the loss
- A Survey on Nonconvex Regularization Based Sparse and Low-Rank Recovery in Signal Processing, Statistics, and Machine Learning