Understanding Self-Attention of Self-Supervised Audio Transformers
arXiv:2006.03265
Abstract
Self-supervised Audio Transformers (SAT) enable great success in many downstream speech applications like ASR, but how they work has not been widely explored yet. In this work, we present multiple strategies for the analysis of attention mechanisms in SAT. We categorize attentions into explainable categories, where we discover each category possesses its own unique functionality. We provide a visualization tool for understanding multi-head self-attention, importance ranking strategies for identifying critical attention, and attention refinement techniques to improve model performance.
Accepted by INTERSPEECH 2020, ICML 2020 Workshop on Self-supervision in Audio and Speech
References in corpus (7)
- Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
- What do you learn from context? Probing for sentence structure in contextualized word representations
- Improving Transformer-based Speech Recognition Using Unsupervised Pre-training
- Effectiveness of self-supervised pre-training for speech recognition
- Self-Attention Attribution: Interpreting Information Interactions Inside Transformer
- Attention Mechanism in Speaker Recognition: What Does It Learn in Deep Speaker Embedding?
- VisBERT: Hidden-State Visualizations for Transformers