The effectiveness of unsupervised subword modeling with autoregressive and cross-lingual phone-aware networks

Abstract

This study addresses unsupervised subword modeling, i.e., learning acoustic feature representations that can distinguish between subword units of a language. We propose a two-stage learning framework that combines self-supervised learning and cross-lingual knowledge transfer. The framework consists of autoregressive predictive coding (APC) as the front-end and a cross-lingual deep neural network (DNN) as the back-end. Experiments on the ABX subword discriminability task conducted with the Libri-light and ZeroSpeech 2017 databases showed that our approach is competitive or superior to state-of-the-art studies. Comprehensive and systematic analyses at the phoneme- and articulatory feature (AF)-level showed that our approach was better at capturing diphthong than monophthong vowel information, while also differences in the amount of information captured for different types of consonants were observed. Moreover, a positive correlation was found between the effectiveness of the back-end in capturing a phoneme's information and the quality of the cross-lingual phone labels assigned to the phoneme. The AF-level analysis together with t-SNE visualization results showed that the proposed approach is better than MFCC and APC features in capturing manner and place of articulation information, vowel height, and backness information. Taken together, the analyses showed that the two stages in our approach are both effective in capturing phoneme and AF information. Nevertheless, monophthong vowel information is less well captured than consonant information, which suggests that future research should focus on improving capturing monophthong vowel information.

18 pages (including 1 page as supplementary material), 13 figures. Accepted for publication in IEEE Open Journal of Signal Processing (OJ-SP)

References in corpus (8)

• Libri-Light: A Benchmark for ASR with Limited or No Supervision
• Unsupervised speech representation learning using WaveNet autoencoders
• Combining Adversarial Training and Disentangled Speech Representation for Robust Zero-Resource Subword Modeling
• Unsupervised feature learning for speech using correspondence and Siamese networks
• Multilingual and Unsupervised Subword Modeling for Zero-Resource Languages
• Improving Unsupervised Subword Modeling via Disentangled Speech Representation Learning and Transformation
• Exploiting Cross-Lingual Speaker and Phonetic Diversity for Unsupervised Subword Modeling
• Unsupervised Subword Modeling Using Autoregressive Pretraining and Cross-Lingual Phone-Aware Modeling