Pay Attention when Required

Abstract

Transformer-based models consist of interleaved feed-forward blocks - that capture content meaning, and relatively more expensive self-attention blocks - that capture context meaning. In this paper, we explored trade-offs and ordering of the blocks to improve upon the current Transformer architecture and proposed PAR Transformer. It needs 35% lower compute time than Transformer-XL achieved by replacing ~63% of the self-attention blocks with feed-forward blocks, and retains the perplexity on WikiText-103 language modelling benchmark. We further validated our results on text8 and enwiki8 datasets, as well as on the BERT model.

9 pages, 5 figures, 7 tables

References in corpus (6)

• Neural Architecture Search with Reinforcement Learning
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Pointer Sentinel Mixture Models
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• Augmenting Self-attention with Persistent Memory
• Finding Fast Transformers: One-Shot Neural Architecture Search by Component Composition

Cited by in corpus (2)

• Revisiting Simple Neural Probabilistic Language Models
• Can the Transformer Be Used as a Drop-in Replacement for RNNs in Text-Generating GANs?