Phonetic-and-Semantic Embedding of Spoken Words with Applications in Spoken Content Retrieval
arXiv:1807.08089
Abstract
Word embedding or Word2Vec has been successful in offering semantics for text words learned from the context of words. Audio Word2Vec was shown to offer phonetic structures for spoken words (signal segments for words) learned from signals within spoken words. This paper proposes a two-stage framework to perform phonetic-and-semantic embedding on spoken words considering the context of the spoken words. Stage 1 performs phonetic embedding with speaker characteristics disentangled. Stage 2 then performs semantic embedding in addition. We further propose to evaluate the phonetic-and-semantic nature of the audio embeddings obtained in Stage 2 by parallelizing with text embeddings. In general, phonetic structure and semantics inevitably disturb each other. For example the words "brother" and "sister" are close in semantics but very different in phonetic structure, while the words "brother" and "bother" are in the other way around. But phonetic-and-semantic embedding is attractive, as shown in the initial experiments on spoken document retrieval. Not only spoken documents including the spoken query can be retrieved based on the phonetic structures, but spoken documents semantically related to the query but not including the query can also be retrieved based on the semantics.
arXiv commentAccepted at SLT2018
References in corpus (22)
- Distributed Representations of Words and Phrases and their Compositionality
- Sequence to Sequence Learning with Neural Networks
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
- Character-Aware Neural Language Models
- Deep Convolutional Inverse Graphics Network
- Effective Approaches to Attention-based Neural Machine Translation
- Domain Separation Networks
- Enriching Word Vectors with Subword Information
- Word Translation Without Parallel Data
- Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data
- A segmental framework for fully-unsupervised large-vocabulary speech recognition
- Unsupervised Adaptation with Domain Separation Networks for Robust Speech Recognition
- Improved Transition-Based Parsing by Modeling Characters instead of Words with LSTMs
- Discriminative Acoustic Word Embeddings: Recurrent Neural Network-Based Approaches
- End-to-End Neural Segmental Models for Speech Recognition
- Non-Adversarial Unsupervised Word Translation
- Character Composition Model with Convolutional Neural Networks for Dependency Parsing on Morphologically Rich Languages
- Towards Unsupervised Automatic Speech Recognition Trained by Unaligned Speech and Text only
- Segmental Audio Word2Vec: Representing Utterances as Sequences of Vectors with Applications in Spoken Term Detection
- Unsupervised Learning of Semantic Audio Representations
- Parsing Speech: A Neural Approach to Integrating Lexical and Acoustic-Prosodic Information
- Learning to Embed Words in Context for Syntactic Tasks