Input-independent Attention Weights Are Expressive Enough: A Study of Attention in Self-supervised Audio Transformers

Tsung-Han Wu, Chun-Chen Hsieh, Yen-Hao Chen +2

In this paper, we seek solutions for reducing the computation complexity of transformer-based models for speech representation learning. We evaluate 10 attention algorithms; then,…