The Singular Values of Convolutional Layers

Abstract

We characterize the singular values of the linear transformation associated with a standard 2D multi-channel convolutional layer, enabling their efficient computation. This characterization also leads to an algorithm for projecting a convolutional layer onto an operator-norm ball. We show that this is an effective regularizer; for example, it improves the test error of a deep residual network using batch normalization on CIFAR-10 from 6.2\% to 5.3\%.

arXiv commentPublished as a conference paper at ICLR 2019

Cited by in corpus (39)

• What is being transferred in transfer learning?
• Invertible Residual Networks
• Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data
• Layer-adaptive sparsity for the Magnitude-based Pruning
• Sorting out Lipschitz function approximation
• SIMBA: Scalable Inversion in Optical Tomography using Deep Denoising Priors
• ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reduction
• Few-Shot Adaptation of Generative Adversarial Networks
• Convolutional Proximal Neural Networks and Plug-and-Play Algorithms
• Parseval Proximal Neural Networks
• The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descent
• Globally-Robust Neural Networks
• Understanding Generalization in Deep Learning via Tensor Methods
• On Mixup Regularization
• Orthogonal Deep Neural Networks
• Fantastic Four: Differentiable Bounds on Singular Values of Convolution Layers
• Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training
• Truth or Backpropaganda? An Empirical Investigation of Deep Learning Theory
• Orthogonal Convolutional Neural Networks
• Effects of Loss Functions And Target Representations on Adversarial Robustness
• MP3net: coherent, minute-long music generation from raw audio with a simple convolutional GAN
• Asymptotic Singular Value Distribution of Linear Convolutional Layers
• Scalable Plug-and-Play ADMM with Convergence Guarantees
• Understanding and Training Deep Diagonal Circulant Neural Networks
• Norm-based generalisation bounds for multi-class convolutional neural networks
• In Search of Robust Measures of Generalization
• Refining the Structure of Neural Networks Using Matrix Conditioning
• A Frobenius norm regularization method for convolutional kernels to avoid unstable gradient problem
• On the Robustness and Generalization of Deep Learning Driven Full Waveform Inversion
• A Peek Into the Hidden Layers of a Convolutional Neural Network Through a Factorization Lens
• Fast Approximate Spectral Normalization for Robust Deep Neural Networks
• On the Structural Sensitivity of Deep Convolutional Networks to the Directions of Fourier Basis Functions
• Infusing Learned Priors into Model-Based Multispectral Imaging
• MSR-DARTS: Minimum Stable Rank of Differentiable Architecture Search
• LEAN: graph-based pruning for convolutional neural networks by extracting longest chains
• Spectral Tensor Train Parameterization of Deep Learning Layers
• EZCrop: Energy-Zoned Channels for Robust Output Pruning
• Boosting the Performance of Plug-and-Play Priors via Denoiser Scaling
• Absum: Simple Regularization Method for Reducing Structural Sensitivity of Convolutional Neural Networks