Block-term Tensor Neural Networks

Abstract

Deep neural networks (DNNs) have achieved outstanding performance in a wide range of applications, e.g., image classification, natural language processing, etc. Despite the good performance, the huge number of parameters in DNNs brings challenges to efficient training of DNNs and also their deployment in low-end devices with limited computing resources. In this paper, we explore the correlations in the weight matrices, and approximate the weight matrices with the low-rank block-term tensors. We name the new corresponding structure as block-term tensor layers (BT-layers), which can be easily adapted to neural network models, such as CNNs and RNNs. In particular, the inputs and the outputs in BT-layers are reshaped into low-dimensional high-order tensors with a similar or improved representation power. Sufficient experiments have demonstrated that BT-layers in CNNs and RNNs can achieve a very large compression ratio on the number of parameters while preserving or improving the representation power of the original DNNs.

12 pages, 15 figures

References in corpus (29)

• Deep Learning in Neural Networks: An Overview
• Distilling the Knowledge in a Neural Network
• Sequence to Sequence Learning with Neural Networks
• Densely Connected Convolutional Networks
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• Compressing Deep Convolutional Networks using Vector Quantization
• Binarized Neural Networks
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Tensorizing Neural Networks
• Compressing Neural Networks with the Hashing Trick
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Action Recognition using Visual Attention
• Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
• Tensor Ring Decomposition
• Unsupervised Generative Modeling Using Matrix Product States
• Era of Big Data Processing: A New Approach via Tensor Networks and Tensor Decompositions
• Exploring Sparsity in Recurrent Neural Networks
• Learning Intrinsic Sparse Structures within Long Short-Term Memory
• Compressing Recurrent Neural Network with Tensor Train
• A Tensorized Transformer for Language Modeling
• Block-Sparse Recurrent Neural Networks
• Long-term Forecasting using Higher Order Tensor RNNs
• TensorNetwork: A Library for Physics and Machine Learning
• Tensor-Train Recurrent Neural Networks for Video Classification
• Sharing Residual Units Through Collective Tensor Factorization in Deep Neural Networks
• Wide Compression: Tensor Ring Nets
• Tensorial Neural Networks: Generalization of Neural Networks and Application to Model Compression
• Tensorial Recurrent Neural Networks for Longitudinal Data Analysis