RealFormer: Transformer Likes Residual Attention
arXiv:2012.11747
Abstract
Transformer is the backbone of modern NLP models. In this paper, we propose RealFormer, a simple and generic technique to create Residual Attention Layer Transformer networks that significantly outperform the canonical Transformer and its variants (BERT, ETC, etc.) on a wide spectrum of tasks including Masked Language Modeling, GLUE, SQuAD, Neural Machine Translation, WikiHop, HotpotQA, Natural Questions, and OpenKP. We also observe empirically that RealFormer stabilizes training and leads to models with sparser attention. Source code and pre-trained checkpoints for RealFormer can be found at https://github.com/google-research/google-research/tree/master/realformer.
arXiv commentFindings of ACL-IJCNLP 2021
References in corpus (11)
- TensorFlow: A system for large-scale machine learning
- Longformer: The Long-Document Transformer
- Linformer: Self-Attention with Linear Complexity
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- Generating Long Sequences with Sparse Transformers
- Reformer: The Efficient Transformer
- Big Bird: Transformers for Longer Sequences
- Towards a Human-like Open-Domain Chatbot
- On Layer Normalization in the Transformer Architecture
- Hopfield Networks is All You Need
- ERNIE 2.0: A Continual Pre-training Framework for Language Understanding
Cited by in corpus (10)
- VARA-TTS: Non-Autoregressive Text-to-Speech Synthesis based on Very Deep VAE with Residual Attention
- Analogous to Evolutionary Algorithm: Designing a Unified Sequence Model
- TRIG: Transformer-Based Text Recognizer with Initial Embedding Guidance
- Evolving Attention with Residual Convolutions
- Attention-based multi-channel speaker verification with ad-hoc microphone arrays
- MedGPT: Medical Concept Prediction from Clinical Narratives
- Transformer-based end-to-end speech recognition with residual Gaussian-based self-attention
- MNet-Sim: A Multi-layered Semantic Similarity Network to Evaluate Sentence Similarity
- Cascaded Head-colliding Attention
- Modeling Concentrated Cross-Attention for Neural Machine Translation with Gaussian Mixture Model