MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark
arXiv:2008.09335
Abstract
Scaling semantic parsing models for task-oriented dialog systems to new languages is often expensive and time-consuming due to the lack of available datasets. Available datasets suffer from several shortcomings: a) they contain few languages b) they contain small amounts of labeled examples per language c) they are based on the simple intent and slot detection paradigm for non-compositional queries. In this paper, we present a new multilingual dataset, called MTOP, comprising of 100k annotated utterances in 6 languages across 11 domains. We use this dataset and other publicly available datasets to conduct a comprehensive benchmarking study on using various state-of-the-art multilingual pre-trained models for task-oriented semantic parsing. We achieve an average improvement of +6.3 points on Slot F1 for the two existing multilingual datasets, over best results reported in their experiments. Furthermore, we demonstrate strong zero-shot performance using pre-trained models combined with automatic translation and alignment, and a proposed distant supervision method to reduce the noise in slot label projection.
13 pages, 2 figures, Accepted at EACL 2021
References in corpus (10)
- A Structured Self-attentive Sentence Embedding
- Exploiting Similarities among Languages for Machine Translation
- Multilingual Denoising Pre-training for Neural Machine Translation
- XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
- Massively Multilingual Word Embeddings
- Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
- Multilingual Translation with Extensible Multilingual Pretraining and Finetuning
- Pre-training via Paraphrasing
- Cross-lingual Retrieval for Iterative Self-Supervised Training
- Massively Multilingual Neural Machine Translation
Cited by in corpus (4)
- Reinforced Iterative Knowledge Distillation for Cross-Lingual Named Entity Recognition
- X2Parser: Cross-Lingual and Cross-Domain Framework for Task-Oriented Compositional Semantic Parsing
- nmT5 -- Is parallel data still relevant for pre-training massively multilingual language models?
- Intent Classification and Slot Filling for Privacy Policies