Discriminative Acoustic Word Embeddings: Recurrent Neural Network-Based Approaches

Abstract

Acoustic word embeddings --- fixed-dimensional vector representations of variable-length spoken word segments --- have begun to be considered for tasks such as speech recognition and query-by-example search. Such embeddings can be learned discriminatively so that they are similar for speech segments corresponding to the same word, while being dissimilar for segments corresponding to different words. Recent work has found that acoustic word embeddings can outperform dynamic time warping on query-by-example search and related word discrimination tasks. However, the space of embedding models and training approaches is still relatively unexplored. In this paper we present new discriminative embedding models based on recurrent neural networks (RNNs). We consider training losses that have been successful in prior work, in particular a cross entropy loss for word classification and a contrastive loss that explicitly aims to separate same-word and different-word pairs in a "Siamese network" training setting. We find that both classifier-based and Siamese RNN embeddings improve over previously reported results on a word discrimination task, with Siamese RNNs outperforming classification models. In addition, we present analyses of the learned embeddings and the effects of variables such as dimensionality and network structure.

arXiv commentTo appear at SLT 2016

References in corpus (3)

• Adam: A Method for Stochastic Optimization
• ADADELTA: An Adaptive Learning Rate Method
• Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition

Cited by in corpus (18)

• Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spaces
• An Unsupervised Autoregressive Model for Speech Representation Learning
• Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech
• Unsupervised End-to-End Learning of Discrete Linguistic Units for Voice Conversion
• Learning Word Embeddings from Speech
• Spoken Term Detection and Relevance Score Estimation using Dot-Product of Pronunciation Embeddings
• Improved Audio Embeddings by Adjacency-Based Clustering with Applications in Spoken Term Detection
• Improving automated segmentation of radio shows with audio embeddings
• Learning Acoustic Word Embeddings with Temporal Context for Query-by-Example Speech Search
• Acoustic Word Embedding System for Code-Switching Query-by-example Spoken Term Detection
• DP-Parse: Finding Word Boundaries from Raw Speech with an Instance Lexicon
• Multilingual Jointly Trained Acoustic and Written Word Embeddings
• Contextual Joint Factor Acoustic Embeddings
• Additional Shared Decoder on Siamese Multi-view Encoders for Learning Acoustic Word Embeddings
• Acoustic span embeddings for multilingual query-by-example search
• Acoustically Grounded Word Embeddings for Improved Acoustics-to-Word Speech Recognition
• Semantic query-by-example speech search using visual grounding
• Phonetic-and-Semantic Embedding of Spoken Words with Applications in Spoken Content Retrieval