GLU Variants Improve Transformer
arXiv:2002.05202
Abstract
Gated Linear Units (arXiv:1612.08083) consist of the component-wise product of two linear projections, one of which is first passed through a sigmoid function. Variations on GLU are possible, using different nonlinear (or even linear) functions in place of sigmoid. We test these variants in the feed-forward sublayers of the Transformer (arXiv:1706.03762) sequence-to-sequence model, and find that some of them yield quality improvements over the typically-used ReLU or GELU activations.
References in corpus (6)
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- Language Modeling with Gated Convolutional Networks
- SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
- SQuAD: 100,000+ Questions for Machine Comprehension of Text
- Searching for Activation Functions
- Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
Cited by in corpus (11)
- mT5: A massively multilingual pre-trained text-to-text transformer
- Revisiting Deep Learning Models for Tabular Data
- Charformer: Fast Character Transformers via Gradient-based Subword Tokenization
- Pay Attention to MLPs
- NormFormer: Improved Transformer Pretraining with Extra Normalization
- Primer: Searching for Efficient Transformers for Language Modeling
- ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning
- A Multi-attribute Controllable Generative Model for Histopathology Image Synthesis
- MedGPT: Medical Concept Prediction from Clinical Narratives
- The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalization
- Distributed Deep Learning in Open Collaborations