5 papers
Discovering Interpretable Algorithms by Decompiling Transformers to RASP
Xinting Huang, Aleksandra Bakalova, Satwik Bhattamishra +2
Recent work has shown that the computations of Transformers can be simulated in the RASP family of programming languages. These findings have enabled improved understanding of the…
How Few-Shot Examples Add Up: A Causal Decomposition of Function Vectors in In-Context Learning
Entang Wang, Yiwei Wang, Aleksandra Bakalova +1
In-context learning (ICL) excels at new tasks from minimal examples, yet we still lack a mechanistic explanation of how few-shot prompts shape a model's function vector (FV)--a cau…
Born a Transformer -- Always a Transformer? On the Effect of Pretraining on Architectural Abilities
Mayank Jobanputra, Yana Veitsman, Yash Sarrof +4
Transformers have theoretical limitations in modeling certain sequence-to-sequence tasks, yet it remains largely unclear if these limitations play a role in large-scale pretrained…
Contextualize-then-Aggregate: Circuits for In-Context Learning in Gemma-2 2B
Aleksandra Bakalova, Yana Veitsman, Xinting Huang +1
In-Context Learning (ICL) is an intriguing ability of large language models (LLMs). Despite a substantial amount of work on its behavioral aspects and how it emerges in miniature s…
Large-scale cosmic ray anisotropies with 19 years of data from the Pierre Auger Observatory
The Pierre Auger Collaboration, A. Abdul Halim, P. Abreu +355
Results are presented for the measurement of large-scale anisotropies in the arrival directions of ultra-high-energy cosmic rays detected at the Pierre Auger Observatory during 19…