Zero-Shot Cross-lingual Classification Using Multilingual Neural Machine Translation

Abstract

Transferring representations from large supervised tasks to downstream tasks has shown promising results in AI fields such as Computer Vision and Natural Language Processing (NLP). In parallel, the recent progress in Machine Translation (MT) has enabled one to train multilingual Neural MT (NMT) systems that can translate between multiple languages and are also capable of performing zero-shot translation. However, little attention has been paid to leveraging representations learned by a multilingual NMT system to enable zero-shot multilinguality in other NLP tasks. In this paper, we demonstrate a simple framework, a multilingual Encoder-Classifier, for cross-lingual transfer learning by reusing the encoder from a multilingual NMT system and stitching it with a task-specific classifier component. Our proposed model achieves significant improvements in the English setup on three benchmark tasks - Amazon Reviews, SST and SNLI. Further, our system can perform classification in a new language for which no classification data was seen during training, showing that zero-shot classification is possible and remarkably competitive. In order to understand the underlying factors contributing to this finding, we conducted a series of analyses on the effect of the shared vocabulary, the training data type for NMT, classifier complexity, encoder representation power, and model generalization on zero-shot performance. Our results provide strong evidence that the representations learned from multilingual NMT systems are widely applicable across languages and tasks.

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Hierarchical Question-Image Co-Attention for Visual Question Answering
• A large annotated corpus for learning natural language inference
• What makes ImageNet good for transfer learning?
• BilBOWA: Fast Bilingual Distributed Representations without Word Alignments
• Massively Multilingual Word Embeddings
• Learned in Translation: Contextualized Word Vectors
• Multilingual Models for Compositional Distributed Semantics

Cited by in corpus (23)

• Cross-lingual Language Model Pretraining
• XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
• Massively Multilingual Neural Machine Translation
• Discriminative Nearest Neighbor Few-Shot Intent Detection by Transferring Natural Language Inference
• Cross-Lingual Transfer Learning for Multilingual Task Oriented Dialog
• Cross-lingual Aspect-Based Sentiment Analysis: A Survey on Tasks, Approaches, and Challenges
• Code-switched inspired losses for generic spoken dialog representations
• Explicit Alignment Objectives for Multilingual Bidirectional Encoders
• Modelling Latent Translations for Cross-Lingual Transfer
• Investigating Multilingual NMT Representations at Scale
• End-to-End Slot Alignment and Recognition for Cross-Lingual NLU
• (Self-Attentive) Autoencoder-based Universal Language Representation for Machine Translation
• Are the Multilingual Models Better? Improving Czech Sentiment with Transformers
• Measuring Semantic Abstraction of Multilingual NMT with Paraphrase Recognition and Generation Tasks
• Towards Interlingua Neural Machine Translation
• Improving Cross-Lingual Reading Comprehension with Self-Training
• tmn at SemEval-2023 Task 9: Multilingual Tweet Intimacy Detection using XLM-T, Google Translate, and Ensemble Learning
• MergeDistill: Merging Pre-trained Language Models using Distillation
• Soft Layer Selection with Meta-Learning for Zero-Shot Cross-Lingual Transfer
• Aligning Cross-Lingual Entities with Multi-Aspect Information
• Multilingual Code-Switching for Zero-Shot Cross-Lingual Intent Prediction and Slot Filling
• Machine Translation for Machines: the Sentiment Classification Use Case
• Learning Contextualised Cross-lingual Word Embeddings and Alignments for Extremely Low-Resource Languages Using Parallel Corpora