27 citations · 57 across the 12 of their papers we have counts for
12 papers
GPQA: A Graduate-Level Google-Proof Q&A Benchmark
David Rein, Betty Li Hou, Asa Cooper Stickland +5
We present GPQA, a challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. We ensure that the questions are high-quality…
Debate Helps Supervise Unreliable Experts
Julian Michael, Salsabila Mahdi, David Rein +4
As AI systems are used to answer more difficult questions and potentially help create new knowledge, judging the truthfulness of their outputs becomes more difficult and more impor…
How Abstract Is Linguistic Generalization in Large Language Models? Experiments with Argument Structure
Michael Wilson, Jackson Petty, Robert Frank
Language models are typically evaluated on their success at predicting the distribution of specific words in specific contexts. Yet linguistic knowledge also encodes relationships…
In-context Learning Generalizes, But Not Always Robustly: The Case of Syntax
Aaron Mueller, Albert Webson, Jackson Petty +1
In-context learning (ICL) is now a common method for teaching large language models (LLMs) new tasks: given labeled examples in the input context, the LLM learns to perform the tas…
The Impact of Depth on Compositional Generalization in Transformer Language Models
Jackson Petty, Sjoerd van Steenkiste, Ishita Dasgupta +3
To process novel sentences, language models (LMs) must generalize compositionally -- combine familiar elements in new ways. What aspects of a model's structure promote compositiona…
(QA): Question Answering with Questionable Assumptions
Najoung Kim, Phu Mon Htut, Samuel R. Bowman +1
Naturally occurring information-seeking questions often contain questionable assumptions -- assumptions that are false or unverifiable. Questions containing questionable assumption…
Do Language Models Learn Position-Role Mappings?
Jackson Petty, Michael Wilson, Robert Frank
How is knowledge of position-role mappings in natural language learned? We explore this question in a computational setting, testing whether a variety of well-performing pertained…
Transformers Generalize Linearly
Jackson Petty, Robert Frank
Natural language exhibits patterns of hierarchically governed dependencies, in which relations between words are sensitive to syntactic structure rather than linear ordering. While…
Sequence-to-Sequence Networks Learn the Meaning of Reflexive Anaphora
Robert Frank, Jackson Petty
Reflexive anaphora present a challenge for semantic interpretation: their meaning varies depending on context in a way that appears to require abstract variables. Past work has rai…
Certain Hyperbolic Regular Polygonal Tiles are Isoperimetric
Jack Hirsch, Kevin Li, Jackson Petty +1
The hexagon is the least-perimeter tile in the Euclidean plane. On hyperbolic surfaces, the isoperimetric problem differs for every given area. Cox conjectured that a regular -g…
Optimal monohedral tilings of hyperbolic surfaces
Leonardo Di Giosia, Jahangir Habib, Jack Hirsch +6
The hexagon is the least-perimeter tile in the Euclidean plane for any given area. On hyperbolic surfaces, this "isoperimetric" problem differs for every given area, as solutions d…
The Optimal Double Bubble for Density
Jack Hirsch, Kevin Li, Jackson Petty +1
In 1993 Foisy et al. proved that the optimal Euclidean planar double bubble---the least-perimeter way to enclose and separate two given areas---is three circular arcs meeting at 12…