Bidirectional Attention as a Mixture of Continuous Word Experts

Kevin Christian Wibisono, Yixin Wang

Bidirectional attention $\unicode{x2013}$ composed of self-attention with positional encodings and the masked language model (MLM) objective $\unicode{x2013}$ has emerged as a key…