Speech To Semantics: Improve ASR and NLU Jointly via All-Neural Interfaces

Abstract

We consider the problem of spoken language understanding (SLU) of extracting natural language intents and associated slot arguments or named entities from speech that is primarily directed at voice assistants. Such a system subsumes both automatic speech recognition (ASR) as well as natural language understanding (NLU). An end-to-end joint SLU model can be built to a required specification opening up the opportunity to deploy on hardware constrained scenarios like devices enabling voice assistants to work offline, in a privacy preserving manner, whilst also reducing server costs. We first present models that extract utterance intent directly from speech without intermediate text output. We then present a compositional model, which generates the transcript using the Listen Attend Spell ASR system and then extracts interpretation using a neural NLU model. Finally, we contrast these methods to a jointly trained end-to-end joint SLU model, consisting of ASR and NLU subsystems which are connected by a neural network based interface instead of text, that produces transcripts as well as NLU interpretation. We show that the jointly trained model shows improvements to ASR incorporating semantic information from NLU and also improves NLU by exposing it to ASR confusion encoded in the hidden layer.

arXiv commentProceedings of INTERSPEECH

References in corpus (2)

• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Sequence Transduction with Recurrent Neural Networks

Cited by in corpus (11)

• End-to-End Spoken Language Understanding: Performance analyses of a voice command task in a low resource setting
• Listen with Intent: Improving Speech Recognition with Audio-to-Intent Front-End
• ASR-GLUE: A New Multi-task Benchmark for ASR-Robust Natural Language Understanding
• Multitask Learning for Low Resource Spoken Language Understanding
• Towards Semi-Supervised Semantics Understanding from Speech
• Pre-training for Spoken Language Understanding with Joint Textual and Phonetic Representation Learning
• Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretraining
• Speech2Slot: An End-to-End Knowledge-based Slot Filling from Speech
• Do as I mean, not as I say: Sequence Loss Training for Spoken Language Understanding
• End-to-End Spoken Language Understanding for Generalized Voice Assistants
• FANS: Fusing ASR and NLU for on-device SLU