Learning Joint Acoustic-Phonetic Word Embeddings
arXiv:1908.00493
Abstract
Most speech recognition tasks pertain to mapping words across two modalities: acoustic and orthographic. In this work, we suggest learning encoders that map variable-length, acoustic or phonetic, sequences that represent words into fixed-dimensional vectors in a shared latent space; such that the distance between two word vectors represents how closely the two words sound. Instead of directly learning the distances between word vectors, we employ weak supervision and model a binary classification task to predict whether two inputs, one of each modality, represent the same word given a distance threshold. We explore various deep-learning models, bimodal contrastive losses, and techniques for mining hard negative examples such as the semi-supervised technique of self-labeling. Our best model achieves an score of 0.95 for the binary classification task.
arXiv comment8 pages, 4 figures
References in corpus (9)
- Adam: A Method for Stochastic Optimization
- Scikit-learn: Machine Learning in Python
- Distributed Representations of Words and Phrases and their Compositionality
- Learning Deep Embeddings with Histogram Loss
- A segmental framework for fully-unsupervised large-vocabulary speech recognition
- Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spaces
- Multi-view Recurrent Neural Acoustic Word Embeddings
- Learning Word Embeddings from Speech
- Learning acoustic word embeddings with phonetically associated triplet network