9 citations · 30 across the 19 of their papers we have counts for
3 papers · 1 filter
Large Language Models Develop Belief State Geometry In-Context
Daniel Balcells, Andrew Jun Lee, Chirag Rastogi +3
Large language models (LLMs) trained on next-token prediction exhibit remarkable in-context learning (ICL) abilities, yet the representations that support ICL remain poorly underst…
Tensor Product Representation Probes Reveal Shared Structure Across Linear Directions
Andrew Lee, Fernanda Viégas, Martin Wattenberg
While researchers are finding concepts represented as linear directions in language models, a bag of linear directions fails to capture relational structure. To better understand t…
Decomposing Query-Key Feature Interactions Using Contrastive Covariances
Andrew Lee, Yonatan Belinkov, Fernanda Viégas +1
Despite the central role of attention heads in Transformers, we lack tools to understand why a model attends to a particular token. To address this, we study the query-key (QK) spa…