Stable Rank Normalization for Improved Generalization in Neural Networks and GANs

Abstract

Exciting new work on the generalization bounds for neural networks (NN) given by Neyshabur et al. , Bartlett et al. closely depend on two parameter-depenedent quantities: the Lipschitz constant upper-bound and the stable rank (a softer version of the rank operator). This leads to an interesting question of whether controlling these quantities might improve the generalization behaviour of NNs. To this end, we propose stable rank normalization (SRN), a novel, optimal, and computationally efficient weight-normalization scheme which minimizes the stable rank of a linear operator. Surprisingly we find that SRN, inspite of being non-convex problem, can be shown to have a unique optimal solution. Moreover, we show that SRN allows control of the data-dependent empirical Lipschitz constant, which in contrast to the Lipschitz upper-bound, reflects the true behaviour of a model on a given dataset. We provide thorough analyses to show that SRN, when applied to the linear layers of a NN for classification, provides striking improvements-11.3% on the generalization gap compared to the standard NN along with significant reduction in memorization. When applied to the discriminator of GANs (called SRN-GAN) it improves Inception, FID, and Neural divergence scores on the CIFAR 10/100 and CelebA datasets, while learning mappings with low empirical Lipschitz constants.

arXiv commentAccepted at the International Conference in Learning Representations, 2020, Addis Ababa, Ethiopia

References in corpus (15)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Conditional Generative Adversarial Nets
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Spectral Normalization for Generative Adversarial Networks
• Improved Training of Wasserstein GANs
• A Learned Representation For Artistic Style
• A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
• Sensitivity and Generalization in Neural Networks: an Empirical Study
• Spectral Norm Regularization for Improving the Generalizability of Deep Learning
• On the regularization of Wasserstein GANs
• Do GANs actually learn the distribution? An empirical study
• Hierarchical Implicit Models and Likelihood-Free Variational Inference
• Norm-Based Capacity Control in Neural Networks
• Deterministic PAC-Bayesian generalization bounds for deep networks via generalizing noise-resilience
• Towards GAN Benchmarks Which Require Generalization