Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators

Abstract

This paper presents a novel pre-trained language models (PLM) compression approach based on the matrix product operator (short as MPO) from quantum many-body physics. It can decompose an original matrix into central tensors (containing the core information) and auxiliary tensors (with only a small proportion of parameters). With the decomposed MPO structure, we propose a novel fine-tuning strategy by only updating the parameters from the auxiliary tensors, and design an optimization algorithm for MPO-based approximation over stacked network architectures. Our approach can be applied to the original or the compressed PLMs in a general way, which derives a lighter network and significantly reduces the parameters to be fine-tuned. Extensive experiments have demonstrated the effectiveness of the proposed approach in model compression, especially the reduction in finetuning parameters (91% reduction on average).

arXiv commentAccepted by ACL 2021 main conference

References in corpus (9)

• Entanglement Entropy and Quantum Field Theory
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Linformer: Self-Attention with Linear Complexity
• Matrix product operator representations
• Compressing Recurrent Neural Network with Tensor Train
• Ultimate tensorization: compressing convolutional and FC layers alike
• A Tensorized Transformer for Language Modeling
• Tensorized Embedding Layers for Efficient Model Compression
• Matrix Product States Algorithms and Continuous Systems