Learning Compressed Transforms with Low Displacement Rank
arXiv:1810.02309
Abstract
The low displacement rank (LDR) framework for structured matrices represents a matrix through two displacement operators and a low-rank residual. Existing use of LDR matrices in deep learning has applied fixed displacement operators encoding forms of shift invariance akin to convolutions. We introduce a class of LDR matrices with more general displacement operators, and explicitly learn over both the operators and the low-rank component. This class generalizes several previous constructions while preserving compression and efficient computation. We prove bounds on the VC dimension of multi-layer neural networks with structured weight matrices and show empirically that our compact parameterization can reduce the sample complexity of learning. When replacing weight layers in fully-connected, convolutional, and recurrent neural networks for image classification and language modeling tasks, our new classes exceed the accuracy of existing compression approaches, and on some tasks also outperform general unstructured layers while using more than 20x fewer parameters.
NeurIPS 2018. Code available at https://github.com/HazyResearch/structured-nets
References in corpus (12)
- Distilling the Knowledge in a Neural Network
- Geometric deep learning: going beyond Euclidean data
- Compressing Neural Networks with the Hashing Trick
- Group Equivariant Convolutional Networks
- On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups
- Rotation equivariant vector field networks
- Exploiting Cyclic Symmetry in Convolutional Neural Networks
- Spherical CNNs
- Structured Transforms for Small-Footprint Deep Learning
- Generalization Error of Invariant Classifiers
- Recycling Randomness with Structure for Sublinear time Kernel Expansions
- Non-Parametric Transformation Networks