Speech-language Pre-training for End-to-end Spoken Language Understanding

Abstract

End-to-end (E2E) spoken language understanding (SLU) can infer semantics directly from speech signal without cascading an automatic speech recognizer (ASR) with a natural language understanding (NLU) module. However, paired utterance recordings and corresponding semantics may not always be available or sufficient to train an E2E SLU model in a real production environment. In this paper, we propose to unify a well-optimized E2E ASR encoder (speech) and a pre-trained language model encoder (language) into a transformer decoder. The unified speech-language pre-trained model (SLP) is continually enhanced on limited labeled data from a target domain by using a conditional masked language model (MLM) objective, and thus can effectively generate a sequence of intent, slot type, and slot value for given input speech in the inference. The experimental results on two public corpora show that our approach to E2E SLU is superior to the conventional cascaded method. It also outperforms the present state-of-the-art approaches to E2E SLU with much less paired data.

References in corpus (6)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Attention-Based Models for Speech Recognition
• Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systems
• Few-shot Natural Language Generation for Task-Oriented Dialog
• SpeechBERT: An Audio-and-text Jointly Learned Language Model for End-to-end Spoken Question Answering

Cited by in corpus (3)

• Pre-training for Spoken Language Understanding with Joint Textual and Phonetic Representation Learning
• Speech2Slot: An End-to-End Knowledge-based Slot Filling from Speech
• Exploring Teacher-Student Learning Approach for Multi-lingual Speech-to-Intent Classification