Universal Transformers

Abstract

Recurrent neural networks (RNNs) sequentially process data by updating their state with each new data point, and have long been the de facto choice for sequence modeling tasks. However, their inherently sequential computation makes them slow to train. Feed-forward and convolutional architectures have recently been shown to achieve superior results on some sequence modeling tasks such as machine translation, with the added advantage that they concurrently process all inputs in the sequence, leading to easy parallelization and faster training times. Despite these successes, however, popular feed-forward sequence models like the Transformer fail to generalize in many simple tasks that recurrent models handle with ease, e.g. copying strings or even simple logical inference when the string or formula lengths exceed those observed at training time. We propose the Universal Transformer (UT), a parallel-in-time self-attentive recurrent sequence model which can be cast as a generalization of the Transformer model and which addresses these issues. UTs combine the parallelizability and global receptive field of feed-forward sequence models like the Transformer with the recurrent inductive bias of RNNs. We also add a dynamic per-position halting mechanism and find that it improves accuracy on several tasks. In contrast to the standard Transformer, under certain assumptions, UTs can be shown to be Turing-complete. Our experiments show that UTs outperform standard Transformers on a wide range of algorithmic and language understanding tasks, including the challenging LAMBADA language modeling task where UTs achieve a new state of the art, and machine translation where UTs achieve a 0.9 BLEU improvement over Transformers on the WMT14 En-De dataset.

arXiv commentPublished at ICLR2019

References in corpus (5)

• Sequence to Sequence Learning with Neural Networks
• End-To-End Memory Networks
• Neural GPUs Learn Algorithms
• Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets
• Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes

Cited by in corpus (80)

• Language Models are Few-Shot Learners
• Pre-trained Models for Natural Language Processing: A Survey
• Scaling Laws for Neural Language Models
• Language Modeling with Deep Transformers
• Theoretical Limitations of Self-Attention in Neural Sequence Models
• TinyBERT: Distilling BERT for Natural Language Understanding
• Rethinking Attention with Performers
• DynaBERT: Dynamic BERT with Adaptive Width and Depth
• Few-Shot Learning via Embedding Adaptation with Set-to-Set Functions
• Hopfield Networks is All You Need
• Explicit Sparse Transformer: Concentrated Attention Through Explicit Selection
• On Identifiability in Transformers
• Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth
• On the Turing Completeness of Modern Neural Network Architectures
• Depth-Adaptive Transformer
• Choose a Transformer: Fourier or Galerkin
• Multiscale Deep Equilibrium Models
• BinaryBERT: Pushing the Limit of BERT Quantization
• BERT Loses Patience: Fast and Robust Inference with Early Exit
• Seq2Slate: Re-ranking and Slate Optimization with RNNs
• Dynamic Neural Networks: A Survey
• Parameter Efficient Multimodal Transformers for Video Representation Learning
• Investigating the Limitations of Transformers with Simple Arithmetic Tasks
• Educating Text Autoencoders: Latent Representation Guidance via Denoising
• Compositional Generalization by Learning Analytical Expressions
• Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers
• Knowledge Enhanced Pretrained Language Models: A Compreshensive Survey
• Empathetic Response Generation through Graph-based Multi-hop Reasoning on Emotional Causality
• Lessons on Parameter Sharing across Layers in Transformers
• An Explicitly Relational Neural Network Architecture
• The Temporal Structure of Language Processing in the Human Brain Corresponds to The Layered Hierarchy of Deep Language Models
• Implicit Feature Pyramid Network for Object Detection
• M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining
• A Survey on Document-level Neural Machine Translation: Methods and Evaluation
• Show Your Work: Scratchpads for Intermediate Computation with Language Models
• SetRank: Learning a Permutation-Invariant Ranking Model for Information Retrieval
• Conditional Self-Attention for Query-based Summarization
• SALSA-TEXT : self attentive latent space based adversarial text generation
• Attention-Informed Mixed-Language Training for Zero-shot Cross-lingual Task-oriented Dialogue Systems
• Early Exiting with Ensemble Internal Classifiers
• An Ensemble of Knowledge Sharing Models for Dynamic Hand Gesture Recognition
• Transferring Inductive Biases through Knowledge Distillation
• Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks
• Controlling Computation versus Quality for Neural Sequence Models
• ODE Transformer: An Ordinary Differential Equation-Inspired Model for Neural Machine Translation
• Normalized Attention Without Probability Cage
• An Efficient Transformer Decoder with Compressed Sub-layers
• Tunable Synaptic Working Memory with Volatile Memristive Devices
• Attention is all you need for Videos: Self-attention based Video Summarization using Universal Transformers
• Soft Prompt Decoding for Multilingual Dense Retrieval
• Object Detection in Video with Spatial-temporal Context Aggregation
• Hard but Robust, Easy but Sensitive: How Encoder and Decoder Perform in Neural Machine Translation
• The Benchmark Lottery
• Towards Automatic Generation of Questions from Long Answers
• TR-BERT: Dynamic Token Reduction for Accelerating BERT Inference
• Distributed Associative Memory Network with Memory Refreshing Loss
• Set-to-Sequence Methods in Machine Learning: a Review
• SoDA: Multi-Object Tracking with Soft Data Association
• Universal Transforming Geometric Network
• Tag-less Back-Translation
• The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalization
• Hybrid Neural Models For Sequence Modelling: The Best Of Three Worlds
• Hierarchical Transformer Network for Utterance-level Emotion Recognition
• Scalable Syntax-Aware Language Models Using Knowledge Distillation
• Designing a Prospective COVID-19 Therapeutic with Reinforcement Learning
• Improving Task-Agnostic BERT Distillation with Layer Mapping Search
• A Mixture of $h-1$ Heads is Better than $h$ Heads
• Attention that does not Explain Away
• Probabilistic Transformer: A Probabilistic Dependency Model for Contextual Word Representation
• Depth-Adaptive Graph Recurrent Network for Text Classification
• Probabilistic Attention for Interactive Segmentation
• Compositional Attention: Disentangling Search and Retrieval
• Generative Models for Security: Attacks, Defenses, and Opportunities
• Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformers
• SQALER: Scaling Question Answering by Decoupling Multi-Hop and Logical Reasoning
• Logic and the $2$-Simplicial Transformer
• A Survey on Green Deep Learning
• ParNet: Position-aware Aggregated Relation Network for Image-Text matching
• Lightweight, Dynamic Graph Convolutional Networks for AMR-to-Text Generation
• Ripple Attention for Visual Perception with Sub-quadratic Complexity