Style Attuned Pre-training and Parameter Efficient Fine-tuning for Spoken Language Understanding

Abstract

Neural models have yielded state-of-the-art results in deciphering spoken language understanding (SLU) problems; however, these models require a significant amount of domain-specific labeled examples for training, which is prohibitively expensive. While pre-trained language models like BERT have been shown to capture a massive amount of knowledge by learning from unlabeled corpora and solve SLU using fewer labeled examples for adaption, the encoding of knowledge is implicit and agnostic to downstream tasks. Such encoding results in model inefficiencies in parameter usage: an entirely new model is required for every domain. To address these challenges, we introduce a novel SLU framework, comprising a conversational language modeling (CLM) pre-training task and a light encoder architecture. The CLM pre-training enables networks to capture the representation of the language in conversation style with the presence of ASR errors. The light encoder architecture separates the shared pre-trained networks from the mappings of generally encoded knowledge to specific domains of SLU, allowing for the domain adaptation to be performed solely at the light encoder and thus increasing efficiency. With the framework, we match the performance of state-of-the-art SLU results on Alexa internal datasets and on two public ones (ATIS, SNIPS), adding only 4.4% parameters per task.

arXiv commentAccepted at INTERSPEECH 2020

References in corpus (11)

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Cross-lingual Language Model Pretraining
• REALM: Retrieval-Augmented Language Model Pre-Training
• TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
• Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces
• Parameter-Efficient Transfer Learning for NLP
• TinyBERT: Distilling BERT for Natural Language Understanding
• BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning
• Just ASK: Building an Architecture for Extensible Self-Service Spoken Language Understanding
• Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representation
• Learning Robust Dialog Policies in Noisy Environments

Cited by in corpus (3)

• Towards Semi-Supervised Semantics Understanding from Speech
• Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretraining
• Two-stage Textual Knowledge Distillation for End-to-End Spoken Language Understanding