13 citations · 28 across the 8 of their papers we have counts for
9 papers
Lessons from the Trenches on Reproducible Evaluation of Language Models
Stella Biderman, Hailey Schoelkopf, Lintang Sutawika +27
Reliable evaluation of language models (LMs) remains an open challenge. Re- searchers and engineers face methodological issues such as the sensitivity of models to evaluation setup…
BizBench: A Quantitative Reasoning Benchmark for Business and Finance
Rik Koncel-Kedziorski, Michael Krumdick, Viet Lai +3
Answering questions within business and finance requires reasoning, precision, and a wide-breadth of technical knowledge. Together, these requirements make this domain difficult fo…
Deep Neural Networks Can Learn Generalizable Same-Different Visual Relations
Alexa R. Tartaglini, Sheridan Feucht, Michael A. Lepori +4
Although deep neural networks can achieve human-level performance on many object recognition benchmarks, prior work suggests that these same models fail to learn simple abstract re…
Training Priors Predict Text-To-Image Model Performance
Charles Lovering, Ellie Pavlick
Text-to-image models can often generate some relations, i.e., "astronaut riding horse", but fail to generate other relations composed of the same basic parts, i.e., "horse riding a…
Evaluation Beyond Task Performance: Analyzing Concepts in AlphaZero in Hex
Charles Lovering, Jessica Zosa Forde, George Konidaris +2
AlphaZero, an approach to reinforcement learning that couples neural networks and Monte Carlo tree search (MCTS), has produced state-of-the-art strategies for traditional board gam…
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
BigScience Workshop, :, Teven Le Scao +391
Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions. While these capabilities have led to wi…
Unit Testing for Concepts in Neural Networks
Charles Lovering, Ellie Pavlick
Many complex problems are naturally understood in terms of symbolic concepts. For example, our concept of "cat" is related to our concepts of "ears" and "whiskers" in a non-arbitra…
Self-play for Data Efficient Language Acquisition
Charles Lovering, Ellie Pavlick
When communicating, people behave consistently across conversational roles: People understand the words they say and are able to produce the words they hear. To date, artificial ag…
Does Data Augmentation Improve Generalization in NLP?
Rohan Jha, Charles Lovering, Ellie Pavlick
Neural models often exploit superficial features to achieve good performance, rather than deriving more general features. Overcoming this tendency is a central challenge in areas s…