Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spaces

Abstract

Recent research has shown that word embedding spaces learned from text corpora of different languages can be aligned without any parallel data supervision. Inspired by the success in unsupervised cross-lingual word embeddings, in this paper we target learning a cross-modal alignment between the embedding spaces of speech and text learned from corpora of their respective modalities in an unsupervised fashion. The proposed framework learns the individual speech and text embedding spaces, and attempts to align the two spaces via adversarial training, followed by a refinement procedure. We show how our framework could be used to perform spoken word classification and translation, and the results on these two tasks demonstrate that the performance of our unsupervised alignment approach is comparable to its supervised counterpart. Our framework is especially useful for developing automatic speech recognition (ASR) and speech-to-text translation systems for low- or zero-resource languages, which have little parallel audio-text data for training modern supervised ASR and speech-to-text translation models, but account for the majority of the languages spoken across the world.

Accepted to NIPS 2018. v2 added the majority word baseline results and other minor fixes. arXiv admin note: text overlap with arXiv:1710.04087 by other authors

References in corpus (12)

• Sequence to Sequence Learning with Neural Networks
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Attention-Based Models for Speech Recognition
• Exploiting Similarities among Languages for Machine Translation
• A segmental framework for fully-unsupervised large-vocabulary speech recognition
• Unsupervised word segmentation and lexicon discovery using acoustic word embeddings
• Augmenting Librispeech with French Translations: A Multimodal Corpus for Direct Speech Translation Evaluation
• Discriminative Acoustic Word Embeddings: Recurrent Neural Network-Based Approaches
• Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech
• Audio Word2Vec: Unsupervised Learning of Audio Segment Representations using Sequence-to-sequence Autoencoder
• Learning Word Embeddings from Speech
• Learning Crosslingual Word Embeddings without Bilingual Corpora

Cited by in corpus (24)

• Libri-Light: A Benchmark for ASR with Limited or No Supervision
• Deep Learning Techniques for Future Intelligent Cross-Media Retrieval
• Unsupervised Automatic Speech Recognition: A Review
• Universal Adversarial Audio Perturbations
• SpeechBERT: An Audio-and-text Jointly Learned Language Model for End-to-end Spoken Question Answering
• Unsupervised Multimodal Representation Learning across Medical Images and Reports
• Unsupervised Speech Recognition
• Unsupervised pre-training for sequence to sequence speech recognition
• Representation Learning for Electronic Health Records
• Multimodal Multitask Representation Learning for Pathology Biobank Metadata Prediction
• Completely Unsupervised Speech Recognition By A Generative Adversarial Network Harmonized With Iteratively Refined Hidden Markov Models
• Optimal Unsupervised Domain Translation
• Almost-unsupervised Speech Recognition with Close-to-zero Resource Based on Phonetic Structures Learned from Very Small Unpaired Speech and Text Data
• PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition
• Towards Semi-Supervised Semantics Understanding from Speech
• Mapping Unparalleled Clinical Professional and Consumer Languages with Embedding Alignment
• ST-BERT: Cross-modal Language Model Pre-training For End-to-end Spoken Language Understanding
• Contextual Text Embeddings for Twi
• Improved Speech Representations with Multi-Target Autoregressive Predictive Coding
• Learning Joint Acoustic-Phonetic Word Embeddings
• From Semi-supervised to Almost-unsupervised Speech Recognition with Very-low Resource by Jointly Learning Phonetic Structures from Audio and Text Embeddings
• Semantic query-by-example speech search using visual grounding
• Analyzing autoencoder-based acoustic word embeddings
• Transductive Zero-Shot Learning using Cross-Modal CycleGAN