Spectral Norm Regularization for Improving the Generalizability of Deep Learning

Abstract

We investigate the generalizability of deep learning based on the sensitivity to input perturbation. We hypothesize that the high sensitivity to the perturbation of data degrades the performance on it. To reduce the sensitivity to perturbation, we propose a simple and effective regularization method, referred to as spectral norm regularization, which penalizes the high spectral norm of weight matrices in neural networks. We provide supportive evidence for the abovementioned hypothesis by experimentally confirming that the models trained using spectral norm regularization exhibit better generalizability than other baseline methods.

References in corpus (6)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• The Loss Surfaces of Multilayer Networks
• Towards Deep Neural Network Architectures Robust to Adversarial Examples
• Revisiting Distributed Synchronous SGD
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• Full-Capacity Unitary Recurrent Neural Networks

Cited by in corpus (3)

• ReachNN: Reachability Analysis of Neural-Network Controlled Systems
• Robust Sparse Regularization: Simultaneously Optimizing Neural Network Robustness and Compactness
• On regularization for a convolutional kernel in neural networks