Audio-Linguistic Embeddings for Spoken Sentences
arXiv:1902.07817
Abstract
We propose spoken sentence embeddings which capture both acoustic and linguistic content. While existing works operate at the character, phoneme, or word level, our method learns long-term dependencies by modeling speech at the sentence level. Formulated as an audio-linguistic multitask learning problem, our encoder-decoder model simultaneously reconstructs acoustic and natural language features from audio. Our results show that spoken sentence embeddings outperform phoneme and word-level baselines on speech recognition and emotion recognition tasks. Ablation studies show that our embeddings can better model high-level acoustic concepts while retaining linguistic content. Overall, our work illustrates the viability of generic, multi-modal sentence embeddings for spoken language understanding.
arXiv commentInternational Conference on Acoustics, Speech, and Signal Processing (ICASSP) 2019
References in corpus (10)
- Distributed Representations of Words and Phrases and their Compositionality
- Sequence to Sequence Learning with Neural Networks
- Distributed Representations of Sentences and Documents
- An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
- On the difficulty of training Recurrent Neural Networks
- An Overview of Multi-Task Learning in Deep Neural Networks
- Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
- Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
- Universal Sentence Encoder
- Skip-Thought Vectors