215 citations · 356 across the 7 of their papers we have counts for
21 papers · 1 filter
Follow the Flow: On Information Flow Across Textual Tokens in Text-to-Image Models
Guy Kaplan, Michael Toker, Yuval Reif +2
Text-to-image generation models suffer from alignment problems, where generated images fail to accurately capture the objects and relations in the text prompt. Prior work has focus…
On Pruning State-Space LLMs
Tamer Ghattas, Michael Hassid, Roy Schwartz
Recent work proposed state-space models (SSMs) as an efficient alternative to transformer-based LLMs. Can these models be pruned to further reduce their computation costs? We adapt…
From Tokens to Words: On the Inner Lexicon of LLMs
Guy Kaplan, Matanel Oren, Yuval Reif +1
Natural language is composed of words, but modern large language models (LLMs) process sub-words as input. A natural question raised by this discrepancy is whether LLMs encode word…
How Much Does Attention Actually Attend? Questioning the Importance of Attention in Pretrained Transformers
Michael Hassid, Hao Peng, Daniel Rotem +4
The attention mechanism is considered the backbone of the widely-used Transformer architecture. It contextualizes the input by computing input-specific attention matrices. We find…
On the Limitations of Dataset Balancing: The Lost Battle Against Spurious Correlations
Roy Schwartz, Gabriel Stanovsky
Recent work has shown that deep learning models in NLP are highly sensitive to low-level correlations between simple features and specific output labels, leading to overfitting and…
Data Contamination: From Memorization to Exploitation
Inbal Magar, Roy Schwartz
Pretrained language models are typically trained on massive web-based datasets, which are often "contaminated" with downstream test sets. It is not clear to what extent models expl…