Approximating Lipschitz continuous functions with GroupSort neural networks
arXiv:2006.05254
Abstract
Recent advances in adversarial attacks and Wasserstein GANs have advocated for use of neural networks with restricted Lipschitz constants. Motivated by these observations, we study the recently introduced GroupSort neural networks, with constraints on the weights, and make a theoretical step towards a better understanding of their expressive power. We show in particular how these networks can represent any Lipschitz continuous piecewise linear functions. We also prove that they are well-suited for approximating Lipschitz continuous functions and exhibit upper bounds on both the depth and size. To conclude, the efficiency of GroupSort networks compared with more standard ReLU networks is illustrated in a set of synthetic experiments.
arXiv comment16 pages
References in corpus (8)
- Spectral Normalization for Generative Adversarial Networks
- On the Number of Linear Regions of Deep Neural Networks
- On Convergence and Stability of GANs
- Generalization and Equilibrium in Generative Adversarial Nets (GANs)
- Representation Benefits of Deep Feedforward Networks
- Improving the Improved Training of Wasserstein GANs: A Consistency Term and Its Dual Effect
- Lipschitz Generative Adversarial Nets
- Norm-preserving Orthogonal Permutation Linear Unit Activation Functions (OPLU)