Single Headed Attention RNN: Stop Thinking With Your Head

Stephen Merity

The leading approaches in language modeling are all obsessed with TV shows of my youth - namely Transformers and Sesame Street. Transformers this, Transformers that, and over here…