Adversarial Learning with Contextual Embeddings for Zero-resource Cross-lingual Classification and NER
arXiv:1909.00153
Abstract
Contextual word embeddings (e.g. GPT, BERT, ELMo, etc.) have demonstrated state-of-the-art performance on various NLP tasks. Recent work with the multilingual version of BERT has shown that the model performs very well in zero-shot and zero-resource cross-lingual settings, where only labeled English data is used to finetune the model. We improve upon multilingual BERT's zero-resource cross-lingual performance via adversarial learning. We report the magnitude of the improvement on the multilingual MLDoc text classification and CoNLL 2002/2003 named entity recognition tasks. Furthermore, we show that language-adversarial training encourages BERT to align the embeddings of English documents and their translations, which may be the cause of the observed performance gains.
arXiv commentIn EMNLP 2019
References in corpus (4)
- Cross-lingual Language Model Pretraining
- Weakly Supervised Cross-Lingual Named Entity Recognition via Effective Annotation and Representation Projection
- Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
- Cross-language Learning with Adversarial Neural Networks: Application to Community Question Answering
Cited by in corpus (4)
- Enhancing Answer Boundary Detection for Multilingual Machine Reading Comprehension
- Can Monolingual Pretrained Models Help Cross-Lingual Classification?
- Towards Multi-Sense Cross-Lingual Alignment of Contextual Embeddings
- Leveraging Adversarial Training in Self-Learning for Cross-Lingual Text Classification