Spectral Norm Regularization for Improving the Generalizability of Deep Learning

Abstract

We investigate the generalizability of deep learning based on the sensitivity to input perturbation. We hypothesize that the high sensitivity to the perturbation of data degrades the performance on it. To reduce the sensitivity to perturbation, we propose a simple and effective regularization method, referred to as spectral norm regularization, which penalizes the high spectral norm of weight matrices in neural networks. We provide supportive evidence for the abovementioned hypothesis by experimentally confirming that the models trained using spectral norm regularization exhibit better generalizability than other baseline methods.

References in corpus (6)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• The Loss Surfaces of Multilayer Networks
• Towards Deep Neural Network Architectures Robust to Adversarial Examples
• Revisiting Distributed Synchronous SGD
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• Full-Capacity Unitary Recurrent Neural Networks

Cited by in corpus (26)

• ReachNN: Reachability Analysis of Neural-Network Controlled Systems
• Regularization Methods for Generative Adversarial Networks: An Overview of Recent Studies
• Robust Sparse Regularization: Simultaneously Optimizing Neural Network Robustness and Compactness
• Towards Efficient and Unbiased Implementation of Lipschitz Continuity in GANs
• The coupling effect of Lipschitz regularization in deep neural networks
• Asymptotic Singular Value Distribution of Linear Convolutional Layers
• Deep Learning for Inverse Problems: Bounds and Regularizers
• Generalised Lipschitz Regularisation Equals Distributional Robustness
• Approximated Orthonormal Normalisation in Training Neural Networks
• RoMA: Robust Model Adaptation for Offline Model-based Optimization
• Preprint: Norm Loss: An efficient yet effective regularization method for deep neural networks
• On regularization for a convolutional kernel in neural networks
• Dynamical System Inspired Adaptive Time Stepping Controller for Residual Network Families
• Fast Approximate Spectral Normalization for Robust Deep Neural Networks
• Training Invertible Linear Layers through Rank-One Perturbations
• Neural Abstract Reasoner
• Building Compact and Robust Deep Neural Networks with Toeplitz Matrices
• Identifying and Exploiting Structures for Reliable Deep Learning
• TaylorGAN: Neighbor-Augmented Policy Update for Sample-Efficient Natural Language Generation
• Noise Stability Regularization for Improving BERT Fine-tuning
• Efficient estimates of optimal transport via low-dimensional embeddings
• Model-Aware Regularization For Learning Approaches To Inverse Problems
• Stochastic Whitening Batch Normalization
• Depthwise Separable Convolutions Allow for Fast and Memory-Efficient Spectral Normalization
• Regularization for convolutional kernel tensors to avoid unstable gradient problem in convolutional neural networks
• OffCon${}^3$: What is state of the art anyway?