DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialogue

Abstract

A long-standing goal of task-oriented dialogue research is the ability to flexibly adapt dialogue models to new domains. To progress research in this direction, we introduce DialoGLUE (Dialogue Language Understanding Evaluation), a public benchmark consisting of 7 task-oriented dialogue datasets covering 4 distinct natural language understanding tasks, designed to encourage dialogue research in representation-based transfer, domain adaptation, and sample-efficient task learning. We release several strong baseline models, demonstrating performance improvements over a vanilla BERT architecture and state-of-the-art results on 5 out of 7 tasks, by pre-training on a large open-domain dialogue corpus and task-adaptive self-supervised training. Through the DialoGLUE benchmark, the baseline methods, and our evaluation scripts, we hope to facilitate progress towards the goal of developing more general task-oriented dialogue models.

Benchmark hosted on: https://evalai.cloudcv.org/web/challenges/challenge-page/708/

References in corpus (7)

• The Natural Language Decathlon: Multitask Learning as Question Answering
• Towards a Human-like Open-Domain Chatbot
• A Survey of Available Corpora for Building Data-Driven Dialogue Systems
• A Simple Language Model for Task-Oriented Dialogue
• Multi-lingual Intent Detection and Slot Filling in a Joint BERT-based Model
• EvalAI: Towards Better Evaluation Systems for AI Agents
• Improving Semantic Parsing for Task Oriented Dialog

Cited by in corpus (15)

• KLUE: Korean Language Understanding Evaluation
• CoCo: Controllable Counterfactuals for Evaluating Dialogue State Trackers
• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
• MultiWOZ 2.3: A multi-domain task-oriented dialogue dataset enhanced with annotation corrections and co-reference annotation
• GLGE: A New General Language Generation Evaluation Benchmark
• ProtAugment: Unsupervised diverse short-texts paraphrasing for intent detection meta-learning
• RADDLE: An Evaluation Benchmark and Analysis Platform for Robust Task-oriented Dialog Systems
• Preview, Attend and Review: Schema-Aware Curriculum Learning for Multi-Domain Dialog State Tracking
• Improving Longer-range Dialogue State Tracking
• "How Robust r u?": Evaluating Task-Oriented Dialogue Systems on Spoken Conversations
• Zero-Shot Dialogue State Tracking via Cross-Task Transfer
• ConvFiT: Conversational Fine-Tuning of Pretrained Language Models
• CANDLE: Decomposing Conditional and Conjunctive Queries for Task-Oriented Dialogue Systems
• Few-Shot Intent Detection via Contrastive Pre-Training and Fine-Tuning
• Effectiveness of Pre-training for Few-shot Intent Classification