Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformers

Abstract

Transformers have shown improved performance when compared to previous architectures for sequence processing such as RNNs. Despite their sizeable performance gains, as recently suggested, the model is computationally expensive to train and with a high parameter budget. In light of this, we explore parameter-sharing methods in Transformers with a specific focus on generative models. We perform an analysis of different parameter sharing/reduction methods and develop the Subformer. Our model combines sandwich-style parameter sharing, which overcomes naive cross-layer parameter sharing in generative models, and self-attentive embedding factorization (SAFE). Experiments on machine translation, abstractive summarization and language modeling show that the Subformer can outperform the Transformer even when using significantly fewer parameters.

arXiv commentEMNLP 2021 Findings

References in corpus (10)

• Teaching Machines to Read and Comprehend
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• Q8BERT: Quantized 8Bit BERT
• Universal Transformers
• Reformer: The Efficient Transformer
• Reducing Transformer Depth on Demand with Structured Dropout
• Deep Equilibrium Models
• fairseq: A Fast, Extensible Toolkit for Sequence Modeling
• Lite Transformer with Long-Short Range Attention
• Non-Autoregressive Machine Translation with Disentangled Context Transformer