1 citations · 1 across the 8 of their papers we have counts for
5 papers · 1 filter
Expand Neurons, Not Parameters
Linghao Kong, Inimai Subramanian, Yonadav Shavit +3
This work demonstrates how increasing the number of neurons in a network without increasing its total number of non-zero parameters improves performance. We show that this gain cor…
Negative Pre-activations Differentiate Syntax
Linghao Kong, Angelina Ning, Micah Adler +1
Modern large language models increasingly use smooth activation functions such as GELU or SiLU, allowing negative pre-activations to carry both signal and gradient. Nevertheless, m…
A Capacity-Based Rationale for Multi-Head Attention
Micah Adler
We study the capacity of the self-attention key-query channel: for a fixed budget, how many distinct token-token relations can a single layer reliably encode? We introduce Relation…
The Birth of Knowledge: Emergent Features across Time, Space, and Scale in Large Language Models
Shashata Sawmya, Micah Adler, Nir Shavit
This paper studies the emergence of interpretable categorical features within large language models (LLMs), analyzing their behavior across training checkpoints (time), transformer…
Towards Combinatorial Interpretability of Neural Computation
Micah Adler, Dan Alistarh, Nir Shavit
We introduce combinatorial interpretability, a methodology for understanding neural computation by analyzing the combinatorial structures in the sign-based categorization of a netw…