Single Headed Attention RNN: Stop Thinking With Your Head

Abstract

The leading approaches in language modeling are all obsessed with TV shows of my youth - namely Transformers and Sesame Street. Transformers this, Transformers that, and over here a bonfire worth of GPU-TPU-neuromorphic wafer scale silicon. We opt for the lazy path of old and proven techniques with a fancy crypto inspired acronym: the Single Headed Attention RNN (SHA-RNN). The author's lone goal is to show that the entire field might have evolved a different direction if we had instead been obsessed with a slightly different acronym and slightly different result. We take a previously strong language model based only on boring LSTMs and get it to within a stone's throw of a stone's throw of state-of-the-art byte level language model results on enwik8. This work has undergone no intensive hyperparameter optimization and lived entirely on a commodity desktop machine that made the author's small studio apartment far too warm in the midst of a San Franciscan summer. The final results are achievable in plus or minus 24 hours on a single GPU as the author is impatient. The attention mechanism is also readily extended to large contexts with minimal computation. Take that Sesame Street.

Addition of citations and contextual results (no attention head, single attention head, attention per layer), removal of wordpiece WikiText-103 numbers due to normalization issues, fix of SHA attention figure Q arrow, other minor fixes

References in corpus (5)

• Generating Long Sequences with Sparse Transformers
• Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
• Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
• An Analysis of Neural Language Modeling at Multiple Scales
• Fast Transformer Decoding: One Write-Head is All You Need

Cited by in corpus (14)

• Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning
• Self-attending RNN for Speech Enhancement to Improve Cross-corpus Generalization
• Addressing Some Limitations of Transformers with Feedback Memory
• Improving the Gating Mechanism of Recurrent Neural Networks
• News-Driven Stock Prediction With Attention-Based Noisy Recurrent State Transition
• Dual-path Self-Attention RNN for Real-Time Speech Enhancement
• Automated essay scoring using efficient transformer-based language models
• A Generalizable Approach to Learning Optimizers
• LegaLMFiT: Efficient Short Legal Text Classification with LSTM Language Model Pre-Training
• Understanding Cross-Lingual Syntactic Transfer in Multilingual Recurrent Neural Networks
• 2kenize: Tying Subword Sequences for Chinese Script Conversion
• When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute
• WaLDORf: Wasteless Language-model Distillation On Reading-comprehension
• Online Self-Attentive Gated RNNs for Real-Time Speaker Separation