Joint Text and Label Generation for Spoken Language Understanding

Abstract

Generalization is a central problem in machine learning, especially when data is limited. Using prior information to enforce constraints is the principled way of encouraging generalization. In this work, we propose to leverage the prior information embedded in pretrained language models (LM) to improve generalization for intent classification and slot labeling tasks with limited training data. Specifically, we extract prior knowledge from pretrained LM in the form of synthetic data, which encode the prior implicitly. We fine-tune the LM to generate an augmented language, which contains not only text but also encodes both intent labels and slot labels. The generated synthetic data can be used to train a classifier later. Since the generated data may contain noise, we rephrase the learning from generated data as learning with noisy labels. We then utilize the mixout regularization for the classifier and prove its effectiveness to resist label noise in generated data. Empirically, our method demonstrates superior performance and outperforms the baseline by a large margin.

References in corpus (21)

• Distilling the Knowledge in a Neural Network
• BERTScore: Evaluating Text Generation with BERT
• Understanding deep learning requires rethinking generalization
• Training Convolutional Networks with Noisy Labels
• On Exact Computation with an Infinitely Wide Neural Net
• Regularizing Neural Networks by Penalizing Confident Output Distributions
• Assessing Generative Models via Precision and Recall
• How does Disagreement Help Generalization against Label Corruption?
• Robust Loss Functions under Label Noise for Deep Neural Networks
• Understanding Back-Translation at Scale
• Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models
• A Kernel Theory of Modern Data Augmentation
• Improved Variational Autoencoders for Text Modeling using Dilated Convolutions
• Benchmarking Natural Language Understanding Services for building Conversational Agents
• On Accurate Evaluation of GANs for Language Generation
• Neural Text Generation: Past, Present and Beyond
• CG-BERT: Conditional Text Generation with BERT for Generalized Few-shot Intent Detection
• Unsupervised Pretraining for Sequence to Sequence Learning
• Simple and Effective Regularization Methods for Training on Noisily Labeled Data with Generalization Guarantee
• Bayesian Incremental Learning for Deep Neural Networks
• Few-Shot Learning as Domain Adaptation: Algorithm and Analysis