Spectral Tensor Train Parameterization of Deep Learning Layers

Abstract

We study low-rank parameterizations of weight matrices with embedded spectral properties in the Deep Learning context. The low-rank property leads to parameter efficiency and permits taking computational shortcuts when computing mappings. Spectral properties are often subject to constraints in optimization problems, leading to better models and stability of optimization. We start by looking at the compact SVD parameterization of weight matrices and identifying redundancy sources in the parameterization. We further apply the Tensor Train (TT) decomposition to the compact SVD components, and propose a non-redundant differentiable parameterization of fixed TT-rank tensor manifolds, termed the Spectral Tensor Train Parameterization (STTP). We demonstrate the effects of neural network compression in the image classification setting and both compression and improved training stability in the generative adversarial training setting.

arXiv commentAccepted at AISTATS 2021

References in corpus (13)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Spectral Normalization for Generative Adversarial Networks
• Wide Residual Networks
• Tensorizing Neural Networks
• The Singular Values of Convolutional Layers
• Spectral tensor-train decomposition
• Ultimate tensorization: compressing convolutional and FC layers alike
• Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group
• Tensor-Train Recurrent Neural Networks for Video Classification
• Wide Compression: Tensor Ring Nets
• T-Basis: a Compact Representation for Neural Networks
• Spectral Regularization for Combating Mode Collapse in GANs