Tensor Decomposition for Model Reduction in Neural Networks: A Review

Abstract

Modern neural networks have revolutionized the fields of computer vision (CV) and Natural Language Processing (NLP). They are widely used for solving complex CV tasks and NLP tasks such as image classification, image generation, and machine translation. Most state-of-the-art neural networks are over-parameterized and require a high computational cost. One straightforward solution is to replace the layers of the networks with their low-rank tensor approximations using different tensor decomposition methods. This paper reviews six tensor decomposition methods and illustrates their ability to compress model parameters of convolutional neural networks (CNNs), recurrent neural networks (RNNs) and Transformers. The accuracy of some compressed models can be higher than the original versions. Evaluations indicate that tensor decompositions can achieve significant reductions in model size, run-time and energy consumption, and are well suited for implementing neural networks on edge devices.

arXiv commentIEEE Circuits and Systems Magazine, 2023

References in corpus (10)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Neural Architecture Search with Reinforcement Learning
• Tensor Decomposition for Signal Processing and Machine Learning
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Exploring the Limits of Language Modeling
• Tensor Ring Decomposition
• Dynamic Channel Pruning: Feature Boosting and Suppression
• Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
• Tensorized Embedding Layers for Efficient Model Compression
• Compressing Recurrent Neural Networks Using Hierarchical Tucker Tensor Decomposition

Cited by in corpus (2)

• Intelligently Augmented Contrastive Tensor Factorization: Empowering Multi-dimensional Time Series Classification in Low-Data Environments
• Lightweight Attribute Localizing Models for Pedestrian Attribute Recognition