Symmetric Dot-Product Attention for Efficient Training of BERT Language Models

Martin Courtois, Malte Ostendorff, Leonhard Hennig +1

Initially introduced as a machine translation model, the Transformer architecture has now become the foundation for modern deep learning architecture, with applications in a wide r…