Efficient Proximal Mapping of the 1-path-norm of Shallow Networks
arXiv:2007.01003
Abstract
We demonstrate two new important properties of the 1-path-norm of shallow neural networks. First, despite its non-smoothness and non-convexity it allows a closed form proximal operator which can be efficiently computed, allowing the use of stochastic proximal-gradient-type methods for regularized empirical risk minimization. Second, when the activation functions is differentiable, it provides an upper bound on the Lipschitz constant of the network. Such bound is tighter than the trivial layer-wise product of Lipschitz constants, motivating its use for training networks robust to adversarial perturbations. In practical experiments we illustrate the advantages of using the proximal mapping and we compare the robustness-accuracy trade-off induced by the 1-path-norm, L1-norm and layer-wise constraints on the Lipschitz constant (Parseval networks).
arXiv commentICML 2020. Fabian Latorre, Paul Rolland and Nadav Hallak have contributed equally
References in corpus (10)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Intriguing properties of neural networks
- Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
- A Survey of Model Compression and Acceleration for Deep Neural Networks
- Group Sparse Regularization for Deep Neural Networks
- Learning the Number of Neurons in Deep Networks
- Parseval Networks: Improving Robustness to Adversarial Examples
- Lipschitz constant estimation of Neural Networks via sparse polynomial optimization
- Provably Correct Automatic Subdifferentiation for Qualified Programs