Towards Semi-Supervised Semantics Understanding from Speech

Abstract

Much recent work on Spoken Language Understanding (SLU) falls short in at least one of three ways: models were trained on oracle text input and neglected the Automatics Speech Recognition (ASR) outputs, models were trained to predict only intents without the slot values, or models were trained on a large amount of in-house data. We proposed a clean and general framework to learn semantics directly from speech with semi-supervision from transcribed speech to address these. Our framework is built upon pretrained end-to-end (E2E) ASR and self-supervised language models, such as BERT, and fine-tuned on a limited amount of target SLU corpus. In parallel, we identified two inadequate settings under which SLU models have been tested: noise-robustness and E2E semantics evaluation. We tested the proposed framework under realistic environmental noises and with a new metric, the slots edit F1 score, on two public SLU corpora. Experiments show that our SLU framework with speech as input can perform on par with those with oracle text as input in semantics understanding, while environmental noises are present, and a limited amount of labeled semantics data is available.

arXiv admin note: text overlap with arXiv:2010.13826

References in corpus (28)

• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• A Comparative Study on Transformer vs RNN in Speech Applications
• Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
• TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
• vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations
• Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces
• Improved Noisy Student Training for Automatic Speech Recognition
• Self-Training for End-to-End Speech Recognition
• Improving Transformer-based Speech Recognition Using Unsupervised Pre-training
• Effectiveness of self-supervised pre-training for speech recognition
• Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spaces
• ESPnet: End-to-End Speech Processing Toolkit
• An Unsupervised Autoregressive Model for Speech Representation Learning
• Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representation
• Speech To Semantics: Improve ASR and NLU Jointly via All-Neural Interfaces
• SpeechBERT: An Audio-and-text Jointly Learned Language Model for End-to-end Spoken Question Answering
• Semi-Supervised Speech Recognition via Local Prior Matching
• Iterative Pseudo-Labeling for Speech Recognition
• Contrastive Predictive Coding Based Feature for Automatic Speaker Verification
• End-to-end named entity extraction from speech
• A scalable noisy speech dataset and online subjective test framework
• Gunrock: A Social Bot for Complex and Engaging Long Conversations
• Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech
• CSTNet: Contrastive Speech Translation Network for Self-Supervised Speech Representation Learning
• Semi-supervised ASR by End-to-end Self-training
• Vector-Quantized Autoregressive Predictive Coding
• Style Attuned Pre-training and Parameter Efficient Fine-tuning for Spoken Language Understanding

Cited by in corpus (3)

• Speech2Slot: An End-to-End Knowledge-based Slot Filling from Speech
• An Adapter-Based Unified Model for Multiple Spoken Language Processing Tasks
• Meta learning to classify intent and slot labels with noisy few shot examples