Similarity Analysis of Self-Supervised Speech Representations
arXiv:2010.11481
Abstract
Self-supervised speech representation learning has recently been a prosperous research topic. Many algorithms have been proposed for learning useful representations from large-scale unlabeled data, and their applications to a wide range of speech tasks have also been investigated. However, there has been little research focusing on understanding the properties of existing approaches. In this work, we aim to provide a comparative study of some of the most representative self-supervised algorithms. Specifically, we quantify the similarities between different self-supervised representations using existing similarity measures. We also design probing tasks to study the correlation between the models' pre-training loss and the amount of specific speech information contained in their learned representations. In addition to showing how various self-supervised models behave differently given the same input, our study also finds that the training objective has a higher impact on representation similarity than architectural choices such as building blocks (RNN/Transformer/CNN) and directionality (uni/bidirectional). Our results also suggest that there exists a strong correlation between pre-training loss and downstream performance for some self-supervised algorithms.
arXiv commentAccepted to ICASSP 2021. Supplementary materials available at https://github.com/iamyuanchung/ICASSP21-Similarity-Supplementary
References in corpus (12)
- A Simple Framework for Contrastive Learning of Visual Representations
- SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
- Libri-Light: A Benchmark for ASR with Limited or No Supervision
- Similarity of Neural Network Representations Revisited
- Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
- TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
- SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability
- Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognition
- Improving Transformer-based Speech Recognition Using Unsupervised Pre-training
- Analysis of Predictive Coding Models for Phonemic Representation Learning in Small Datasets
- A Further Study of Unsupervised Pre-training for Transformer Based Speech Recognition
- Unsupervised Subword Modeling Using Autoregressive Pretraining and Cross-Lingual Phone-Aware Modeling