3 papers
cs.LG2025
A Capacity-Based Rationale for Multi-Head Attention
Micah Adler
We study the capacity of the self-attention key-query channel: for a fixed budget, how many distinct token-token relations can a single layer reliably encode? We introduce Relation…
cs.CL2025
The Birth of Knowledge: Emergent Features across Time, Space, and Scale in Large Language Models
Shashata Sawmya, Micah Adler, Nir Shavit
This paper studies the emergence of interpretable categorical features within large language models (LLMs), analyzing their behavior across training checkpoints (time), transformer…
cs.LG2025
Towards Combinatorial Interpretability of Neural Computation
Micah Adler, Dan Alistarh, Nir Shavit
We introduce combinatorial interpretability, a methodology for understanding neural computation by analyzing the combinatorial structures in the sign-based categorization of a netw…