Understanding Self-Attention of Self-Supervised Audio Transformers

Shu-wen Yang, Andy T. Liu, Hung-yi Lee

Self-supervised Audio Transformers (SAT) enable great success in many downstream speech applications like ASR, but how they work has not been widely explored yet. In this work, we…