DIET: Lightweight Language Understanding for Dialogue Systems

Abstract

Large-scale pre-trained language models have shown impressive results on language understanding benchmarks like GLUE and SuperGLUE, improving considerably over other pre-training methods like distributed representations (GloVe) and purely supervised approaches. We introduce the Dual Intent and Entity Transformer (DIET) architecture, and study the effectiveness of different pre-trained representations on intent and entity prediction, two common dialogue language understanding tasks. DIET advances the state of the art on a complex multi-domain NLU dataset and achieves similarly high performance on other simpler datasets. Surprisingly, we show that there is no clear benefit to using large pre-trained models for this task, and in fact DIET improves upon the current state of the art even in a purely supervised setup without any pre-trained embeddings. Our best performing model outperforms fine-tuning BERT and is about six times faster to train.

v3: Updated results for the best model

References in corpus (3)

• DocBERT: BERT for Document Classification
• Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding
• PatentBERT: Patent Classification with Fine-Tuning a pre-trained BERT Model

Cited by in corpus (12)

• Natural Language Understanding for Argumentative Dialogue Systems in the Opinion Building Domain
• STAR: A Schema-Guided Dialog Dataset for Transfer Learning
• UserSimCRS: A User Simulation Toolkit for Evaluating Conversational Recommender Systems
• DRAGON: A Dialogue-Based Robot for Assistive Navigation with Visual Language Grounding
• We are all Individuals: The Role of Robot Personality and Human Traits in Trustworthy Interaction
• A Co-Interactive Transformer for Joint Slot Filling and Intent Detection
• COVID-19 Smart Chatbot Prototype for Patient Monitoring
• Do We Need Online NLU Tools?
• Benchmarking Commercial Intent Detection Services with Practice-Driven Evaluations
• Do as I mean, not as I say: Sequence Loss Training for Spoken Language Understanding
• A Sequence to Sequence Model for Extracting Multiple Product Name Entities from Dialog
• Multi-stage Clarification in Conversational AI: The case of Question-Answering Dialogue Systems