12 papers
MURANO: Design, Run, and Reproduce Mechanistic Interpretability Experiments as Composable Pipelines
Alireza Bayat Makou, Emirhan Böge, Phu Gia Hoang +5
This paper presents Murano, an open source framework for designing, running, and reproducing mechanistic interpretability studies of large language models, intended for researchers…
Sparse Autoencoders Encode Both Concepts and Functions: The Downstream Geometry of Feature Effects
Phu Gia Hoang, Anwoy Chatterjee, Tanmoy Chakraborty +2
The wide-scale use of sparse autoencoders (SAEs) as interpretability tools is limited by inconsistent links between SAE features and model behavior. Features with clear activation…
The FIL Hypothesis: Inductive Biases Help with Kernel Engineering
Nikolai Rozanov, Subhabrata Dutta, Preslav Nakov +1
The Bitter Lesson, which posits that general-purpose methods that scale with computation and data ultimately outperform those with built-in human knowledge, has become a dominant p…
Many Circuits, One Mechanism: Input Variation and Evaluation Granularity in Circuit Discovery
Alireza Bayat Makou, Jingcheng Niu, Subhabrata Dutta +1
Circuit discovery methods identify subgraphs that explain model behaviors, and structural differences between discovered circuits are commonly interpreted as evidence of distinct m…
Subjective Code Preferences in Experts and Large Language Models
Anna Mokhova, Subhabrata Dutta, Iryna Gurevych +1
Large Language Models (LLMs) have become increasingly popular for coding tasks, with subjective coding preferences being an essential element to adapt to programmers' personal need…
Hierarchical Latent Structures in Data Generation Process Unify Mechanistic Phenomena across Scale
Jonas Rohweder, Subhabrata Dutta, Iryna Gurevych
Contemporary studies in mechanistic interpretability have uncovered many puzzling phenomena in the neural information processing of Transformer-based language models, such as induc…
Patches of Nonlinearity: Instruction Vectors in Large Language Models
Irina Bigoulaeva, Jonas Rohweder, Subhabrata Dutta +1
Despite the recent success of instruction-tuned language models and their ubiquitous usage, very little is known of how models process instructions internally. In this work, we add…
Reward Modeling for Scientific Writing Evaluation
Furkan Şahinuç, Subhabrata Dutta, Iryna Gurevych
Scientific writing is an expert-domain task that demands deep domain knowledge, task-specific requirements and reasoning capabilities that leverage the domain knowledge to satisfy…
Expert Preference-based Evaluation of Automated Related Work Generation
Furkan Şahinuç, Subhabrata Dutta, Iryna Gurevych
Expert domain writing, such as scientific writing, typically demands extensive domain knowledge. Although large language models (LLMs) show promising potential in this task, evalua…
Factual Self-Awareness in Language Models: Representation, Robustness, and Scaling
Hovhannes Tamoyan, Subhabrata Dutta, Iryna Gurevych
Factual incorrectness in generated content is one of the primary concerns in ubiquitous deployment of large language models (LLMs). Prior findings suggest LLMs can (sometimes) dete…
Illusion or Algorithm? Investigating Memorization, Emergence, and Symbolic Processing in In-Context Learning
Jingcheng Niu, Subhabrata Dutta, Ahmed Elshabrawy +2
Large-scale Transformer language models (LMs) trained solely on next-token prediction with web-scale data can solve a wide range of tasks after seeing just a few examples. The mech…
Problem Solving Through Human-AI Preference-Based Cooperation
Subhabrata Dutta, Timo Kaufmann, Goran Glavaš +7
While there is a widespread belief that artificial general intelligence (AGI) -- or even superhuman AI -- is imminent, complex problems in expert domains are far from being solved.…