85 citations · 193 across the 8 of their papers we have counts for
5 papers · 1 filter
Understanding In-Context Learning via Supportive Pretraining Data
Xiaochuang Han, Daniel Simig, Todor Mihaylov +3
In-context learning (ICL) improves language models' performance on a variety of NLP tasks by simply demonstrating a handful of examples at inference time. It is not well understood…
Evaluating end-to-end entity linking on domain-specific knowledge bases: Learning about ancient technologies from museum collections
Sebastian Cadavid-Sanchez, Khalil Kacem, Rafael Aparecido Martins Frade +4
To study social, economic, and historical questions, researchers in the social sciences and humanities have started to use increasingly large unstructured textual datasets. While r…
MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers
Lili Yu, Dániel Simig, Colin Flaherty +3
Autoregressive transformers are spectacular models for short sequences but scale poorly to long sequences such as high-resolution images, podcasts, code, or books. We proposed Mega…
SemDeDup: Data-efficient learning at web-scale through semantic deduplication
Amro Abbas, Kushal Tirumala, Dániel Simig +2
Progress in machine learning has been driven in large part by massive increases in data. However, large web-scale datasets such as LAION are largely uncurated beyond searches for e…
OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization
Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru +15
Recent work has shown that fine-tuning large pre-trained language models on a collection of tasks described via instructions, a.k.a. instruction-tuning, improves their zero and few…