215 citations · 516 across the 30 of their papers we have counts for
12 papers · 1 filter
Catwalk: A Unified Language Model Evaluation Framework for Many Datasets
Dirk Groeneveld, Anas Awadalla, Iz Beltagy +7
The success of large language models has shifted the evaluation paradigms in natural language processing (NLP). The community's interest has drifted towards comparing NLP models ac…
Paloma: A Benchmark for Evaluating Language Model Fit
Ian Magnusson, Akshita Bhagia, Valentin Hofmann +13
Evaluations of language models (LMs) commonly report perplexity on monolithic data held out from training. Implicitly or explicitly, this data is composed of domains--varying distr…
The Rise of Open Science: Tracking the Evolution and Perceived Value of Data and Methods Link-Sharing Practices
Hancheng Cao, Jesse Dodge, Kyle Lo +2
In recent years, funding agencies and journals increasingly advocate for open science practices (e.g. data and method sharing) to improve the transparency, access, and reproducibil…
What's In My Big Data?
Yanai Elazar, Akshita Bhagia, Ian Magnusson +10
Large text corpora are the backbone of language models. However, we have a limited understanding of the content of these corpora, including general statistics, quality, social fact…
Language Models Hallucinate, but May Excel at Fact Verification
Jian Guan, Jesse Dodge, David Wadden +2
Recent progress in natural language processing (NLP) owes much to remarkable advances in large language models (LLMs). Nevertheless, LLMs frequently "hallucinate," resulting in non…
Efficiency Pentathlon: A Standardized Arena for Efficiency Evaluation
Hao Peng, Qingqing Cao, Jesse Dodge +11
Rising computational demands of modern natural language processing (NLP) systems have increased the barrier to entry for cutting-edge research while posing serious environmental co…