Initialization and Regularization of Factorized Neural Layers
arXiv:2105.01029
Abstract
Factorized layers--operations parameterized by products of two or more matrices--occur in a variety of deep learning contexts, including compressed model training, certain types of knowledge distillation, and multi-head self-attention architectures. We study how to initialize and regularize deep nets containing such layers, examining two simple, understudied schemes, spectral initialization and Frobenius decay, for improving their performance. The guiding insight is to design optimization routines for these networks that are as close as possible to that of their well-tuned, non-decomposed counterparts; we back this intuition with an analysis of how the initialization and regularization schemes impact training with gradient descent, drawing on modern attempts to understand the interplay of weight-decay and batch-normalization. Empirically, we highlight the benefits of spectral initialization and Frobenius decay across a variety of settings. In model compression, we show that they enable low-rank methods to significantly outperform both unstructured sparsity and tensor methods on the task of training low-memory residual networks; analogs of the schemes also improve the performance of tensor decomposition techniques. For knowledge distillation, Frobenius decay enables a simple, overcomplete baseline that yields a compact model from over-parameterized training without requiring retraining with or pruning a teacher network. Finally, we show how both schemes applied to multi-head attention lead to improved performance on both translation and unsupervised pre-training.
arXiv commentICLR 2021 camera-ready, amended due to error pointed out in arXiv:2209.13569v1 (amendment shown in blue)
References in corpus (19)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Decoupled Weight Decay Regularization
- Convolutional Sequence to Sequence Learning
- SNIP: Single-shot Network Pruning based on Connection Sensitivity
- Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science
- Convolutional neural networks with low-rank regularization
- A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
- Comparing Rewinding and Fine-tuning in Neural Network Pruning
- Deep Rewiring: Training very sparse deep networks
- Picking Winning Tickets Before Training by Preserving Gradient Flow
- Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization
- Norm matters: efficient and accurate normalization schemes in deep networks
- ExpandNets: Linear Over-parameterization to Train Compact Convolutional Networks
- Deeper Insights into Weight Sharing in Neural Architecture Search
- Width Provably Matters in Optimization for Deep Linear Neural Networks
- EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis
- ACDC: A Structured Efficient Linear Layer
- Separable Layers Enable Structured Efficient Linear Substitutions