Cross-Lingual Transfer Learning for Multilingual Task Oriented Dialog

Abstract

One of the first steps in the utterance interpretation pipeline of many task-oriented conversational AI systems is to identify user intents and the corresponding slots. Since data collection for machine learning models for this task is time-consuming, it is desirable to make use of existing data in a high-resource language to train models in low-resource languages. However, development of such models has largely been hindered by the lack of multilingual training data. In this paper, we present a new data set of 57k annotated utterances in English (43k), Spanish (8.6k) and Thai (5k) across the domains weather, alarm, and reminder. We use this data set to evaluate three different cross-lingual transfer methods: (1) translating the training data, (2) using cross-lingual pre-trained embeddings, and (3) a novel method of using a multilingual machine translation encoder as contextual word representations. We find that given several hundred training examples in the the target language, the latter two methods outperform translating the training data. Further, in very low-resource settings, multilingual contextual word representations give better results than using cross-lingual static embeddings. We also compare the cross-lingual methods to using monolingual resources in the form of contextual ELMo representations and find that given just small amounts of target language data, this method outperforms all cross-lingual methods, which highlights the need for more sophisticated cross-lingual methods.

arXiv comment11 pages, to be presented at NAACL 2019

References in corpus (9)

• Convolutional Sequence to Sequence Learning
• A Structured Self-attentive Sentence Embedding
• Word Translation Without Parallel Data
• Transfer Learning for Sequence Tagging with Hierarchical Recurrent Networks
• XNLI: Evaluating Cross-lingual Sentence Representations
• Zero-Shot Cross-lingual Classification Using Multilingual Neural Machine Translation
• Vocabulary Selection Strategies for Neural Machine Translation
• PyText: A Seamless Path from NLP research to production
• Neural Cross-Lingual Named Entity Recognition with Minimal Resources

Cited by in corpus (7)

• Recent Advances and Challenges in Task-oriented Dialog System
• Simple is Better! Lightweight Data Augmentation for Low Resource Slot Filling and Intent Classification
• Out-of-domain Detection for Natural Language Understanding in Dialog Systems
• Cross-Lingual Transfer Learning for Question Answering
• Augmented Natural Language for Generative Sequence Labeling
• A Joint and Domain-Adaptive Approach to Spoken Language Understanding
• Leveraging Adversarial Training in Self-Learning for Cross-Lingual Text Classification