Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretraining
arXiv:2010.13826
Abstract
Much recent work on Spoken Language Understanding (SLU) is limited in at least one of three ways: models were trained on oracle text input and neglected ASR errors, models were trained to predict only intents without the slot values, or models were trained on a large amount of in-house data. In this paper, we propose a clean and general framework to learn semantics directly from speech with semi-supervision from transcribed or untranscribed speech to address these issues. Our framework is built upon pretrained end-to-end (E2E) ASR and self-supervised language models, such as BERT, and fine-tuned on a limited amount of target SLU data. We study two semi-supervised settings for the ASR component: supervised pretraining on transcribed speech, and unsupervised pretraining by replacing the ASR encoder with self-supervised speech representations, such as wav2vec. In parallel, we identify two essential criteria for evaluating SLU models: environmental noise-robustness and E2E semantics evaluation. Experiments on ATIS show that our SLU framework with speech as input can perform on par with those using oracle text as input in semantics understanding, even though environmental noise is present and a limited amount of labeled semantics data is available for training.
References in corpus (10)
- SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
- Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
- Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces
- An Unsupervised Autoregressive Model for Speech Representation Learning
- Speech To Semantics: Improve ASR and NLU Jointly via All-Neural Interfaces
- SpeechBERT: An Audio-and-text Jointly Learned Language Model for End-to-end Spoken Question Answering
- End-to-end named entity extraction from speech
- A scalable noisy speech dataset and online subjective test framework
- Gunrock: A Social Bot for Complex and Engaging Long Conversations
- Style Attuned Pre-training and Parameter Efficient Fine-tuning for Spoken Language Understanding