7 citations · 14 across the 22 of their papers we have counts for
23 papers
Adversarial Concept Search: Predicting Compositional Errors From Feature Geometry
Jennifer Meng Lu, Ruochen Zhang, Isabelle Lee +3
Humans cannot always intuit what scenarios are most challenging to LLMs. Hoping to capture challenging edge cases, developers either design problems to be difficult for humans or c…
Handling and Interpreting Missing Modalities in Patient Clinical Trajectories via Autoregressive Sequence Modeling
Andrew Wang, Ellie Pavlick, Ritambhara Singh
An active challenge in developing multimodal machine learning (ML) models for healthcare is handling missing modalities during training and deployment. As clinical datasets are inh…
Source-Modality Monitoring in Vision-Language Models
Etha Tianze Hua, Tian Yun, Ellie Pavlick
We define and investigate source-modality monitoring -- the ability of multimodal models to track and communicate the input source from which pieces of information originate. We co…
Shared Lexical Task Representations Explain Behavioral Variability In LLMs
Zhuonan Yang, Jacob Xiaochen Li, Francisco Piedrahita Velez +5
One of the most common complaints about large language models (LLMs) is their prompt sensitivity -- that is, the fact that their ability to perform a task or provide a correct answ…
Video Finetuning Improves Reasoning Between Frames
Ruiqi Yang, Tian Yun, Zihan Wang +1
Multimodal large language models (LLMs) have made rapid progress in visual understanding, yet their extension from images to videos often reduces to a naive concatenation of frame…
Can LLMs subtract numbers?
Mayank Jobanputra, Nils Philipp Walter, Maitrey Mehta +7
We present a systematic study of subtraction in large language models (LLMs). While prior benchmarks emphasize addition and multiplication, subtraction has received comparatively l…
How Do Language Models Compose Functions?
Apoorv Khandelwal, Ellie Pavlick
While large language models (LLMs) appear to be increasingly capable of solving compositional tasks, it is an open question whether they do so using compositional mechanisms. In th…
From Prediction to Understanding: Will AI Foundation Models Transform Brain Science?
Thomas Serre, Ellie Pavlick
Generative pretraining (the "GPT" in ChatGPT) enables language models to learn from vast amounts of internet text without human supervision. This approach has driven breakthroughs…
What is an "Abstract Reasoner"? Revisiting Experiments and Arguments about Large Language Models
Tian Yun, Chen Sun, Ellie Pavlick
Recent work has argued that large language models (LLMs) are not "abstract reasoners", citing their poor zero-shot performance on a variety of challenging tasks as evidence. We rev…
LLMs model how humans induce logically structured rules
Alyssa Loo, Ellie Pavlick, Roman Feiman
A central goal of cognitive science is to provide a computationally explicit account of both the structure of the mind and its development: what are the primitive representational…
How Do Vision-Language Models Process Conflicting Information Across Modalities?
Tianze Hua, Tian Yun, Ellie Pavlick
AI models are increasingly required to be multimodal, integrating disparate input streams into a coherent state representation on which subsequent behaviors and actions can be base…
Is This Just Fantasy? Language Model Representations Reflect Human Judgments of Event Plausibility
Michael A. Lepori, Jennifer Hu, Ishita Dasgupta +3
Language models (LMs) are used for a diverse range of tasks, from question answering to writing fantastical stories. In order to reliably accomplish these tasks, LMs must be able t…
Paths Not Taken: Understanding and Mending the Multilingual Factual Recall Pipeline
Meng Lu, Ruochen Zhang, Carsten Eickhoff +1
Multilingual large language models (LLMs) often exhibit factual inconsistencies across languages, with significantly better performance in factual recall tasks in English than in o…
Born a Transformer -- Always a Transformer? On the Effect of Pretraining on Architectural Abilities
Mayank Jobanputra, Yana Veitsman, Yash Sarrof +4
Transformers have theoretical limitations in modeling certain sequence-to-sequence tasks, yet it remains largely unclear if these limitations play a role in large-scale pretrained…
Does Training on Synthetic Data Make Models Less Robust?
Lingze Zhang, Ellie Pavlick
An increasingly common practice is to train large language models (LLMs) using synthetic data. Often this synthetic data is produced by the same or similar LLMs as those it is bein…
A Knapsack by Any Other Name: Presentation impacts LLM performance on NP-hard problems
Alex Duchnowski, Ellie Pavlick, Alexander Koller
To investigate the effect of problem presentation on LLMs' ability to solve optimization problems, we introduce the dataset of Everyday Hard Optimization Problems (EHOP), a collect…
Mitigating Extrinsic Gender Bias for Bangla Classification Tasks
Sajib Kumar Saha Joy, Arman Hassan Mahy, Meherin Sultana +4
In this study, we investigate extrinsic gender bias in Bangla pretrained language models, a largely underexplored area in low-resource languages. To assess this bias, we construct…
LLMs as Models for Analogical Reasoning
Sam Musker, Alex Duchnowski, Raphaël Millière +1
Analogical reasoning -- the capacity to identify and map structural relationships between different domains -- is fundamental to human cognition and learning. Recent studies have s…
Lessons from the Trenches on Reproducible Evaluation of Language Models
Stella Biderman, Hailey Schoelkopf, Lintang Sutawika +27
Reliable evaluation of language models (LMs) remains an open challenge. Re- searchers and engineers face methodological issues such as the sensitivity of models to evaluation setup…
mOthello: When Do Cross-Lingual Representation Alignment and Cross-Lingual Transfer Emerge in Multilingual Models?
Tianze Hua, Tian Yun, Ellie Pavlick
Many pretrained multilingual models exhibit cross-lingual transfer ability, which is often attributed to a learned language-neutral representation during pretraining. However, it r…
Bayesian Preference Elicitation with Language Models
Kunal Handa, Yarin Gal, Ellie Pavlick +4
Aligning AI systems to users' interests requires understanding and incorporating humans' complex values and preferences. Recently, language models (LMs) have been used to gather in…
Understanding Subjectivity through the Lens of Motivational Context in Model-Generated Image Satisfaction
Senjuti Dutta, Sherol Chen, Sunny Mak +7
Image generation models are poised to become ubiquitous in a range of applications. These models are often fine-tuned and evaluated using human quality judgments that assume a univ…
The dynamic interplay between in-context and in-weight learning in humans and neural networks
Jacob Russin, Ellie Pavlick, Michael J. Frank
Human learning embodies a striking duality: sometimes, we appear capable of following logical, compositional rules and benefit from structured curricula (e.g., in formal education)…