352 citations · 459 across the 9 of their papers we have counts for
3 papers · 1 filter
Input-independent Attention Weights Are Expressive Enough: A Study of Attention in Self-supervised Audio Transformers
Tsung-Han Wu, Chun-Chen Hsieh, Yen-Hao Chen +2
In this paper, we seek solutions for reducing the computation complexity of transformer-based models for speech representation learning. We evaluate 10 attention algorithms; then,…
Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representation
Po-Han Chi, Pei-Hung Chung, Tsung-Han Wu +4
For self-supervised speech processing, it is crucial to use pretrained models as speech representation extractors. In recent works, increasing the size of the model has been utiliz…
BERT's output layer recognizes all hidden layers? Some Intriguing Phenomena and a simple way to boost BERT
Wei-Tsung Kao, Tsung-Han Wu, Po-Han Chi +2
Although Bidirectional Encoder Representations from Transformers (BERT) have achieved tremendous success in many natural language processing (NLP) tasks, it remains a black box. A…