Probing Acoustic Representations for Phonetic Properties

Abstract

Pre-trained acoustic representations such as wav2vec and DeCoAR have attained impressive word error rates (WER) for speech recognition benchmarks, particularly when labeled data is limited. But little is known about what phonetic properties these various representations acquire, and how well they encode transferable features of speech. We compare features from two conventional and four pre-trained systems in some simple frame-level phonetic classification tasks, with classifiers trained on features from one version of the TIMIT dataset and tested on features from another. All contextualized representations offered some level of transferability across domains, and models pre-trained on more audio data give better results; but overall, DeCoAR, the system with the simplest architecture, performs best. This type of benchmarking analysis can thus uncover relative strengths of various proposed acoustic representations.

References in corpus (12)

• Scikit-learn: Machine Learning in Python
• Distributed Representations of Words and Phrases and their Compositionality
• Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
• vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations
• fairseq: A Fast, Extensible Toolkit for Sequence Modeling
• Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognition
• CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordings
• DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization
• Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representation
• BERTphone: Phonetically-Aware Encoder Representations for Utterance-Level Speaker and Language Recognition
• Speech-XLNet: Unsupervised Acoustic Model Pretraining For Self-Attention Networks
• What all do audio transformer models hear? Probing Acoustic Representations for Language Delivery and its Structure