Spectral Norm Regularization for Improving the Generalizability of Deep Learning

Abstract

We investigate the generalizability of deep learning based on the sensitivity to input perturbation. We hypothesize that the high sensitivity to the perturbation of data degrades the performance on it. To reduce the sensitivity to perturbation, we propose a simple and effective regularization method, referred to as spectral norm regularization, which penalizes the high spectral norm of weight matrices in neural networks. We provide supportive evidence for the abovementioned hypothesis by experimentally confirming that the models trained using spectral norm regularization exhibit better generalizability than other baseline methods.

References in corpus (9)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Densely Connected Convolutional Networks
• Exploring the Limits of Language Modeling
• The Loss Surfaces of Multilayer Networks
• Towards Deep Neural Network Architectures Robust to Adversarial Examples
• Revisiting Distributed Synchronous SGD
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• Distributed Deep Learning Using Synchronous Stochastic Gradient Descent
• Full-Capacity Unitary Recurrent Neural Networks

Cited by in corpus (74)

• Spectral Normalization for Generative Adversarial Networks
• NVAE: A Deep Hierarchical Variational Autoencoder
• Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy
• PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization
• Review: Deep Learning in Electron Microscopy
• Hybrid Spectrogram and Waveform Source Separation
• Sorting out Lipschitz function approximation
• Music Demixing Challenge 2021
• Robust Learning with Jacobian Regularization
• NAIS-Net: Stable Deep Networks from Non-Autonomous Differential Equations
• Adversarially Robust Neural Architectures
• L2-Nonexpansive Neural Networks
• Analysis of Generalizability of Deep Neural Networks Based on the Complexity of Decision Boundary
• Why Spectral Normalization Stabilizes GANs: Analysis and Improvements
• ReachNN: Reachability Analysis of Neural-Network Controlled Systems
• Smoothed Geometry for Robust Attribution
• Regularization Methods for Generative Adversarial Networks: An Overview of Recent Studies
• Lipschitz regularized Deep Neural Networks generalize and are adversarially robust
• Why Does Little Robustness Help? A Further Step Towards Understanding Adversarial Transferability
• Generative Adversarial Networks Applied to Synthetic Financial Scenarios Generation
• Improving Pre-trained Language Model Fine-tuning with Noise Stability Regularization
• Robust Sparse Regularization: Simultaneously Optimizing Neural Network Robustness and Compactness
• Robust learning with implicit residual networks
• A case for new neural network smoothness constraints
• Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?
• Wide-minima Density Hypothesis and the Explore-Exploit Learning Rate Schedule
• Selecting Relevant Features from a Multi-domain Representation for Few-shot Classification
• Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspective
• The Low-Rank Simplicity Bias in Deep Networks
• An Empirical Study of Large-Batch Stochastic Gradient Descent with Structured Covariance Noise
• Towards Efficient and Unbiased Implementation of Lipschitz Continuity in GANs
• Unrolled Wirtinger Flow with Deep Decoding Priors for Phaseless Imaging
• Stable and expressive recurrent vision models
• Lipschitz Networks and Distributional Robustness
• The coupling effect of Lipschitz regularization in deep neural networks
• Boosting the Certified Robustness of L-infinity Distance Nets
• Open Event Extraction from Online Text using a Generative Adversarial Network
• Asymptotic Singular Value Distribution of Linear Convolutional Layers
• Deep Learning for Inverse Problems: Bounds and Regularizers
• Generalised Lipschitz Regularisation Equals Distributional Robustness
• An FPGA-Based On-Device Reinforcement Learning Approach using Online Sequential Learning
• In Search of Robust Measures of Generalization
• A Frobenius norm regularization method for convolutional kernels to avoid unstable gradient problem
• Towards Certifying L-infinity Robustness using Neural Networks with L-inf-dist Neurons
• Improve Generalization and Robustness of Neural Networks via Weight Scale Shifting Invariant Regularizations
• Optimization-Inspired Learning with Architecture Augmentations and Control Mechanisms for Low-Level Vision
• A Novel Framework for Selection of GANs for an Application
• Pores for thought: The use of generative adversarial networks for the stochastic reconstruction of 3D multi-phase electrode microstructures with periodic boundaries
• Approximated Orthonormal Normalisation in Training Neural Networks
• Stable Rank Normalization for Improved Generalization in Neural Networks and GANs
• RoMA: Robust Model Adaptation for Offline Model-based Optimization
• On the Robustness and Generalization of Deep Learning Driven Full Waveform Inversion
• Training Invertible Linear Layers through Rank-One Perturbations
• Preprint: Norm Loss: An efficient yet effective regularization method for deep neural networks
• Fast Approximate Spectral Normalization for Robust Deep Neural Networks
• Learning Lipschitz Feedback Policies from Expert Demonstrations: Closed-Loop Guarantees, Generalization and Robustness
• An automatic differentiation system for the age of differential privacy
• Dynamical System Inspired Adaptive Time Stepping Controller for Residual Network Families
• Neural Abstract Reasoner
• Building Compact and Robust Deep Neural Networks with Toeplitz Matrices
• Linearly Constrained Weights: Reducing Activation Shift for Faster Training of Neural Networks
• On regularization for a convolutional kernel in neural networks
• Model-Aware Regularization For Learning Approaches To Inverse Problems
• Identifying and Exploiting Structures for Reliable Deep Learning
• Depthwise Separable Convolutions Allow for Fast and Memory-Efficient Spectral Normalization
• OffCon${}^3$: What is state of the art anyway?
• Stochastic Whitening Batch Normalization
• Regularization for convolutional kernel tensors to avoid unstable gradient problem in convolutional neural networks
• Learning with Hyperspherical Uniformity
• Recent Advances in Large Margin Learning
• Absum: Simple Regularization Method for Reducing Structural Sensitivity of Convolutional Neural Networks
• TaylorGAN: Neighbor-Augmented Policy Update for Sample-Efficient Natural Language Generation
• Efficient estimates of optimal transport via low-dimensional embeddings
• Noise Stability Regularization for Improving BERT Fine-tuning