Saccade Attention Networks: Using Transfer Learning of Attention to Reduce Network Sizes

Marc Estafanous

One of the limitations of transformer networks is the sequence length due to the quadratic nature of the attention matrix. Classical self attention uses the entire sequence length,…