Input-independent Attention Weights Are Expressive Enough: A Study of Attention in Self-supervised Audio Transformers
arXiv:2006.05174
Abstract
In this paper, we seek solutions for reducing the computation complexity of transformer-based models for speech representation learning. We evaluate 10 attention algorithms; then, we pre-train the transformer-based model with those attention algorithms in a self-supervised fashion and treat them as feature extractors on downstream tasks, including phoneme classification and speaker classification. With the assistance of t-SNE, PCA and some observation, the attention weights in self-supervised audio transformers can be categorized into four general cases. Based on these cases and some analyses, we are able to use a specific set of attention weights to initialize the model. Our approach shows comparable performance to the typical self-attention yet requires 20% less time in both training and inference.
References in corpus (15)
- Representation Learning with Contrastive Predictive Coding
- WaveNet: A Generative Model for Raw Audio
- A Tutorial on Principal Component Analysis
- Longformer: The Long-Document Transformer
- A Structured Self-attentive Sentence Embedding
- Generating Long Sequences with Sparse Transformers
- Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
- Synthesizer: Rethinking Self-Attention in Transformer Models
- Lite Transformer with Long-Short Range Attention
- Improving Transformer-based Speech Recognition Using Unsupervised Pre-training
- Effectiveness of self-supervised pre-training for speech recognition
- On Symmetric and Asymmetric LSHs for Inner Product Search
- Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representation
- Improved Asymmetric Locality Sensitive Hashing (ALSH) for Maximum Inner Product Search (MIPS)
- Speech-XLNet: Unsupervised Acoustic Model Pretraining For Self-Attention Networks