A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matters
arXiv:2012.15682
Abstract
Few-shot crosslingual transfer has been shown to outperform its zero-shot counterpart with pretrained encoders like multilingual BERT. Despite its growing popularity, little to no attention has been paid to standardizing and analyzing the design of few-shot experiments. In this work, we highlight a fundamental risk posed by this shortcoming, illustrating that the model exhibits a high degree of sensitivity to the selection of few shots. We conduct a large-scale experimental study on 40 sets of sampled few shots for six diverse NLP tasks across up to 40 languages. We provide an analysis of success and failure cases of few-shot transfer, which highlights the role of lexical features. Additionally, we show that a straightforward full model finetuning approach is quite effective for few-shot transfer, outperforming several state-of-the-art few-shot approaches. As a step towards standardizing few-shot crosslingual experimental designs, we make our sampled few shots publicly available.
ACL-IJCNLP 2021
References in corpus (14)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Shortcut Learning in Deep Neural Networks
- Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection
- Frustratingly Simple Few-Shot Object Detection
- Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
- Learning and Evaluating General Linguistic Intelligence
- Multilingual Alignment of Contextual Word Representations
- Making Pre-trained Language Models Better Few-shot Learners
- Meta-learning for Few-shot Natural Language Processing: A Survey
- Transfer Learning and Distant Supervision for Multilingual Transformer Models: A Study on African Languages
- A Call for More Rigor in Unsupervised Cross-lingual Learning
- Inducing Language-Agnostic Multilingual Representations
- FewJoint: A Few-shot Learning Benchmark for Joint Language Understanding