85 citations · 193 across the 8 of their papers we have counts for
9 papers
Understanding In-Context Learning via Supportive Pretraining Data
Xiaochuang Han, Daniel Simig, Todor Mihaylov +3
In-context learning (ICL) improves language models' performance on a variety of NLP tasks by simply demonstrating a handful of examples at inference time. It is not well understood…
Evaluating end-to-end entity linking on domain-specific knowledge bases: Learning about ancient technologies from museum collections
Sebastian Cadavid-Sanchez, Khalil Kacem, Rafael Aparecido Martins Frade +4
To study social, economic, and historical questions, researchers in the social sciences and humanities have started to use increasingly large unstructured textual datasets. While r…
MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers
Lili Yu, Dániel Simig, Colin Flaherty +3
Autoregressive transformers are spectacular models for short sequences but scale poorly to long sequences such as high-resolution images, podcasts, code, or books. We proposed Mega…
SemDeDup: Data-efficient learning at web-scale through semantic deduplication
Amro Abbas, Kushal Tirumala, Dániel Simig +2
Progress in machine learning has been driven in large part by massive increases in data. However, large web-scale datasets such as LAION are largely uncurated beyond searches for e…
OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization
Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru +15
Recent work has shown that fine-tuning large pre-trained language models on a collection of tasks described via instructions, a.k.a. instruction-tuning, improves their zero and few…
Text Characterization Toolkit
Daniel Simig, Tianlu Wang, Verna Dankers +4
In NLP, models are usually evaluated by reporting single-number performance scores on a number of readily available benchmarks, without much deeper analysis. Here, we argue that -…
OPT: Open Pre-trained Transformer Language Models
Susan Zhang, Stephen Roller, Naman Goyal +16
Large language models, which are often trained for hundreds of thousands of compute days, have shown remarkable capabilities for zero- and few-shot learning. Given their computatio…
Open Vocabulary Extreme Classification Using Generative Models
Daniel Simig, Fabio Petroni, Pouya Yanki +4
The extreme multi-label classification (XMC) task aims at tagging content with a subset of labels from an extremely large label set. The label vocabulary is typically defined in ad…
Few-shot Learning with Multilingual Language Models
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe +18
Large-scale generative language models such as GPT-3 are competitive few-shot learners. While these models are known to be able to jointly represent many different languages, their…