Depthwise Separable Convolutions Allow for Fast and Memory-Efficient Spectral Normalization
arXiv:2102.06496
Abstract
An increasing number of models require the control of the spectral norm of convolutional layers of a neural network. While there is an abundance of methods for estimating and enforcing upper bounds on those during training, they are typically costly in either memory or time. In this work, we introduce a very simple method for spectral normalization of depthwise separable convolutions, which introduces negligible computational and memory overhead. We demonstrate the effectiveness of our method on image classification tasks using standard architectures like MobileNetV2.
References in corpus (5)
- NICE: Non-linear Independent Components Estimation
- Spectral Norm Regularization for Improving the Generalizability of Deep Learning
- The Reversible Residual Network: Backpropagation Without Storing Activations
- Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networks
- Lipschitz constant estimation of Neural Networks via sparse polynomial optimization