Towards Semi-Supervised Semantics Understanding from Speech

Abstract

Much recent work on Spoken Language Understanding (SLU) falls short in at least one of three ways: models were trained on oracle text input and neglected the Automatics Speech Recognition (ASR) outputs, models were trained to predict only intents without the slot values, or models were trained on a large amount of in-house data. We proposed a clean and general framework to learn semantics directly from speech with semi-supervision from transcribed speech to address these. Our framework is built upon pretrained end-to-end (E2E) ASR and self-supervised language models, such as BERT, and fine-tuned on a limited amount of target SLU corpus. In parallel, we identified two inadequate settings under which SLU models have been tested: noise-robustness and E2E semantics evaluation. We tested the proposed framework under realistic environmental noises and with a new metric, the slots edit F1 score, on two public SLU corpora. Experiments show that our SLU framework with speech as input can perform on par with those with oracle text as input in semantics understanding, while environmental noises are present, and a limited amount of labeled semantics data is available.

arXiv admin note: text overlap with arXiv:2010.13826

References in corpus (10)

• vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations
• Improving Transformer-based Speech Recognition Using Unsupervised Pre-training
• An Unsupervised Autoregressive Model for Speech Representation Learning
• Semi-Supervised Speech Recognition via Local Prior Matching
• Iterative Pseudo-Labeling for Speech Recognition
• Contrastive Predictive Coding Based Feature for Automatic Speaker Verification
• Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech
• Gunrock: A Social Bot for Complex and Engaging Long Conversations
• CSTNet: Contrastive Speech Translation Network for Self-Supervised Speech Representation Learning
• Vector-Quantized Autoregressive Predictive Coding

Cited by in corpus (1)

• Meta learning to classify intent and slot labels with noisy few shot examples