Theoretical Properties for Neural Networks with Weight Matrices of Low Displacement Rank

Abstract

Recently low displacement rank (LDR) matrices, or so-called structured matrices, have been proposed to compress large-scale neural networks. Empirical results have shown that neural networks with weight matrices of LDR matrices, referred as LDR neural networks, can achieve significant reduction in space and computational complexity while retaining high accuracy. We formally study LDR matrices in deep learning. First, we prove the universal approximation property of LDR neural networks with a mild condition on the displacement operators. We then show that the error bounds of LDR neural networks are as efficient as general neural networks with both single-layer and multiple-layer structure. Finally, we propose back-propagation based training algorithm for general LDR neural networks.

13 pages, 1 figure

References in corpus (5)

• Sequence to Sequence Learning with Neural Networks
• Compressing Deep Convolutional Networks using Vector Quantization
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Learning Structured Sparsity in Deep Neural Networks
• Benefits of depth in neural networks

Cited by in corpus (7)

• CirCNN: Accelerating and Compressing Deep Neural Networks Using Block-CirculantWeight Matrices
• A Unified Framework of DNN Weight Pruning and Weight Clustering/Quantization Using ADMM
• C-LSTM: Enabling Efficient LSTM using Structured Compression Techniques on FPGAs
• ProdSumNet: reducing model parameters in deep neural networks via product-of-sums matrix decompositions
• On the Universal Approximation Property and Equivalence of Stochastic Computing-based Neural Networks and Binary Neural Networks
• E-RNN: Design Optimization for Efficient Recurrent Neural Networks in FPGAs
• CircConv: A Structured Convolution with Low Complexity