4 papers
Causal and Interpretable Structures in LLM Compositional Tasks
Gurbir Arora, Toni J. B. Liu, Jiajun Bao +2
Large language models are able to solve tasks whose answers depend on not only individual input tokens, but also on relations among them. How is such relational information represe…
The Direction of Ignorance: LLMs Encode and Temper a Unigram Prior
Toni J. B. Liu, Jiajun Bao, Yizhou Liu +5
What does a language model predict when it has few clues?The answer lies in its unembedding geometry: a single direction of the unembedding matrix encodes the unigram distribution…
A Graph Signal Processing Perspective on Numerical Sequence Representations in LLM In-Context Learning
Jiajun Bao, Zihao Qi, Toni J. B. Liu +4
Pretrained large language models (LLMs) have demonstrated in-context learning (ICL) capabilities for numerical inference over sequences serialized as text. Prior work has identifie…
Jacobian Scopes: A Unified Geometric Framework for Token-Level LLM Attribution
Toni J. B. Liu, Baran Zadeoğlu, Nicolas Boullé +3
Large language models (LLMs) make next-token predictions based on clues present in their context, such as semantic descriptions and in-context examples. Yet, elucidating which prio…