Towards Unsupervised Automatic Speech Recognition Trained by Unaligned Speech and Text only

Abstract

Automatic speech recognition (ASR) has been widely researched with supervised approaches, while many low-resourced languages lack audio-text aligned data, and supervised methods cannot be applied on them. In this work, we propose a framework to achieve unsupervised ASR on a read English speech dataset, where audio and text are unaligned. In the first stage, each word-level audio segment in the utterances is represented by a vector representation extracted by a sequence-of-sequence autoencoder, in which phonetic information and speaker information are disentangled. Secondly, semantic embeddings of audio segments are trained from the vector representations using a skip-gram model. Last but not the least, an unsupervised method is utilized to transform semantic embeddings of audio segments to text embedding space, and finally the transformed embeddings are mapped to words. With the above framework, we are towards unsupervised ASR trained by unaligned text and speech only.

arXiv commentCode is released: https://github.com/grtzsohalf/Towards-Unsupervised-ASR

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• Improved Training of Wasserstein GANs
• Word Translation Without Parallel Data
• Unsupervised Machine Translation Using Monolingual Corpora Only
• Unsupervised Adaptation with Domain Separation Networks for Robust Speech Recognition
• How to evaluate word embeddings? On importance of data efficiency and simple supervised tasks
• Gate Activation Signal Analysis for Gated Recurrent Neural Networks and Its Correlation with Phoneme Boundaries
• Non-Adversarial Unsupervised Word Translation

Cited by in corpus (6)

• Almost-unsupervised Speech Recognition with Close-to-zero Resource Based on Phonetic Structures Learned from Very Small Unpaired Speech and Text Data
• LRSpeech: Extremely Low-Resource Speech Synthesis and Recognition
• Improving Word Recognition in Speech Transcriptions by Decision-level Fusion of Stemming and Two-way Phoneme Pruning
• Improved Audio Embeddings by Adjacency-Based Clustering with Applications in Spoken Term Detection
• Unsupervised Speech Recognition via Segmental Empirical Output Distribution Matching
• Phonetic-and-Semantic Embedding of Spoken Words with Applications in Spoken Content Retrieval