A Tensorized Transformer for Language Modeling

Abstract

Latest development of neural models has connected the encoder and decoder through a self-attention mechanism. In particular, Transformer, which is solely based on self-attention, has led to breakthroughs in Natural Language Processing (NLP) tasks. However, the multi-head attention mechanism, as a key component of Transformer, limits the effective deployment of the model to a resource-limited setting. In this paper, based on the ideas of tensor decomposition and parameters sharing, we propose a novel self-attention model (namely Multi-linear attention) with Block-Term Tensor Decomposition (BTD). We test and verify the proposed attention method on three language modeling tasks (i.e., PTB, WikiText-103 and One-billion) and a neural machine translation task (i.e., WMT-2016 English-German). Multi-linear attention can not only largely compress the model parameters but also obtain performance improvements, compared with a number of language modeling approaches, such as Transformer, Transformer-XL, and Transformer with tensor train decomposition.

Accepted by NeurIPS 2019

Cited by in corpus (32)

• TinyBERT: Distilling BERT for Natural Language Understanding
• DynaBERT: Dynamic BERT with Adaptive Width and Depth
• Training with Quantization Noise for Extreme Model Compression
• Convolutional Tensor-Train LSTM for Spatio-temporal Learning
• Encoding word order in complex embeddings
• Is Attention Better Than Matrix Decomposition?
• Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT
• Tensorized Embedding Layers for Efficient Model Compression
• BinaryBERT: Pushing the Limit of BERT Quantization
• Block-Term Tensor Decomposition: Model Selection and Computation
• AdaBERT: Task-Adaptive BERT Compression with Differentiable Neural Architecture Search
• AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuning
• PowerNorm: Rethinking Batch Normalization in Transformers
• KDLSQ-BERT: A Quantized Bert Combining Knowledge Distillation with Learned Step Size Quantization
• TernaryBERT: Distillation-aware Ultra-low Bit BERT
• DeFormer: Decomposing Pre-trained Transformers for Faster Question Answering
• Adversarial Training with Contrastive Learning in NLP
• TensorCoder: Dimension-Wise Attention via Tensor Representation for Natural Language Modeling
• Redesigning the Transformer Architecture with Insights from Multi-particle Dynamical Systems
• TENT: Tensorized Encoder Transformer for Temperature Forecasting
• SuperShaper: Task-Agnostic Super Pre-training of BERT Models with Variable Hidden Dimensions
• Improving Task-Agnostic BERT Distillation with Layer Mapping Search
• Semantic Representation and Inference for NLP
• Building Compact and Robust Deep Neural Networks with Toeplitz Matrices
• Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators
• Block-term Tensor Neural Networks
• Multi-Head Self-Attention with Role-Guided Masks
• Learning Hard Retrieval Decoder Attention for Transformers
• MARS: Masked Automatic Ranks Selection in Tensor Decompositions
• Layer Reduction: Accelerating Conformer-Based Self-Supervised Model via Layer Consistency
• A Survey on Green Deep Learning
• Paradigm Shift in Language Modeling: Revisiting CNN for Modeling Sanskrit Originated Bengali and Hindi Language