33 citations · 127 across the 31 of their papers we have counts for
36 papers
What LLMs explain is not what they believe: Evaluating explanation sufficiency under models' own input beliefs
Nhi Nguyen, Shauli Ravfogel, Rajesh Ranganath
Large language models (LLMs) are increasingly deployed in high-stakes domains, where free-text explanations such as chain-of-thought and post-hoc rationales are used to justify mod…
Geometric Factual Recall in Transformers
Shauli Ravfogel, Gilad Yehudai, Joan Bruna +1
How do transformer language models memorize factual associations? A common view casts internal weight matrices as associative memories over pairs of embeddings, requiring parameter…
Can LLMs Introspect? A Reality Check
Shashwat Singh, Tal Linzen, Shauli Ravfogel
Can large language models detect and report their own internal states? A number of recent studies have argued that they can. Drawing on lessons from human metacognition research, w…
The Truthfulness Spectrum Hypothesis
Zhuofan Josh Ying, Shauli Ravfogel, Nikolaus Kriegeskorte +1
Large language models (LLMs) have been reported to linearly encode truthfulness, yet recent work questions this finding's generality. We reconcile these views with the truthfulness…
Discrete Diffusion Models Exploit Asymmetry to Solve Lookahead Planning Tasks
Itamar Trainin, Shauli Ravfogel, Omri Abend +1
While Autoregressive (AR) Transformer-based Generative Language Models are frequently employed for lookahead tasks, recent research suggests a potential discrepancy in their abilit…
State over Tokens: Characterizing the Role of Reasoning Tokens
Mosh Levy, Zohar Elyoseph, Shauli Ravfogel +1
Large Language Models (LLMs) can generate reasoning tokens before their final answer to boost performance on complex tasks. While these sequences seem like human thought processes,…
From Benchmarks to Skills: Low-Rank Factors for LLM Evaluation
Aviya Maimon, Amir DN Cohen, Gal Vishne +2
Current evaluations of large language models (LLMs) rely heavily on a growing collection of benchmarks and on aggregate benchmark scores, yet it remains unclear what this compariso…
RELIC: Evaluating Complex Reasoning via the Recognition of Languages In-Context
Jackson Petty, Michael Y. Hu, Wentao Wang +3
Large language models (LLMs) are increasingly used to solve complex tasks where they must retrieve and compose many pieces of in-context information in long reasoning chains. For m…
Diversity Over Quantity: A Lesson From Few Shot Relation Classification
Amir DN Cohen, Shauli Ravfogel, Shaltiel Shmidman +1
In few-shot relation classification (FSRC), models must generalize to novel relations with only a few labeled examples. While much of the recent progress in NLP has focused on scal…
Gumbel Counterfactual Generation From Language Models
Shauli Ravfogel, Anej Svete, Vésteinn Snæbjarnarson +1
Understanding and manipulating the causal generation mechanisms in language models is essential for controlling their behavior. Previous work has primarily relied on techniques suc…
GRADE: Quantifying Sample Diversity in Text-to-Image Models
Royi Rassin, Aviv Slobodkin, Shauli Ravfogel +2
We introduce GRADE, an automatic method for quantifying sample diversity in text-to-image models. Our method leverages the world knowledge embedded in large language models and vis…
On Affine Homotopy between Language Encoders
Robin SM Chan, Reda Boumasmoud, Anej Svete +8
Pre-trained language encoders -- functions that represent text as vectors -- are an integral component of many NLP tasks. We tackle a natural question in language encoder analysis:…
A Practical Method for Generating String Counterfactuals
Matan Avitan, Ryan Cotterell, Yoav Goldberg +1
Interventions targeting the representation space of language models (LMs) have emerged as an effective means to influence model behavior. Such methods are employed, for example, to…
Representation Surgery: Theory and Practice of Affine Steering
Shashwat Singh, Shauli Ravfogel, Jonathan Herzig +3
Language models often exhibit undesirable behavior, e.g., generating toxic or gender-biased text. In the case of neural language models, an encoding of the undesirable behavior is…
Linguistic Binding in Diffusion Models: Enhancing Attribute Correspondence through Attention Map Alignment
Royi Rassin, Eran Hirsch, Daniel Glickman +3
Text-conditioned image generation models often generate incorrect associations between entities and their visual attributes. This reflects an impaired mapping between linguistic bi…
LEACE: Perfect linear concept erasure in closed form
Nora Belrose, David Schneider-Joseph, Shauli Ravfogel +3
Concept erasure aims to remove specified features from an embedding. It can improve fairness (e.g. preventing a classifier from using gender or race) and interpretability (e.g. rem…
All Roads Lead to Rome? Exploring the Invariance of Transformers' Representations
Yuxin Ren, Qipeng Guo, Zhijing Jin +4
Transformer models bring propelling advances in various NLP tasks, thus inducing lots of interpretability research on the learned representations of the models. However, we raise a…
Conformal Nucleus Sampling
Shauli Ravfogel, Yoav Goldberg, Jacob Goldberger
Language models generate text based on successively sampling the next word. A decoding procedure based on nucleus (top-) sampling chooses from the smallest possible set of words…
Description-Based Text Similarity
Shauli Ravfogel, Valentina Pyatkin, Amir DN Cohen +2
Identifying texts with a given semantics is central for many information seeking scenarios. Similarity search over vector embeddings appear to be central to this ability, yet the s…
DALLE-2 is Seeing Double: Flaws in Word-to-Concept Mapping in Text2Image Models
Royi Rassin, Shauli Ravfogel, Yoav Goldberg
We study the way DALLE-2 maps symbols (words) in the prompt to their references (entities or properties of entities in the generated image). We show that in stark contrast to the w…
Log-linear Guardedness and its Implications
Shauli Ravfogel, Yoav Goldberg, Ryan Cotterell
Methods for erasing human-interpretable concepts from neural representations that assume linearity have been found to be tractable and useful. However, the impact of this removal o…
Analyzing Gender Representation in Multilingual Models
Hila Gonen, Shauli Ravfogel, Yoav Goldberg
Multilingual language models were shown to allow for nontrivial transfer across scripts and languages. In this work, we study the structure of the internal representations that ena…
Kernelized Concept Erasure
Shauli Ravfogel, Francisco Vargas, Yoav Goldberg +1
The representation space of neural models for textual data emerges in an unsupervised manner during training. Understanding how those representations encode human-interpretable con…
Linear Adversarial Concept Erasure
Shauli Ravfogel, Michael Twiton, Yoav Goldberg +1
Modern neural models trained on textual data rely on pre-trained representations that emerge without direct supervision. As these representations are increasingly being used in rea…