1 citations · 1 across the 8 of their papers we have counts for
7 papers · 1 filter
Attention Mean Fields Predict Average Representation Dynamics and Reveal Context-Specific Computation
Micah Adler, John W. Byers, Mark Crovella
A language model's representation geometry is not predetermined; it evolves as the model runs. A faithful account of that geometry must capture that dynamic process, and so cannot…
The Sparsity Whisperer
Linghao Kong, Inimai Subramanian, Micah Adler +3
Pruning reduces the inference cost of large language models, but existing criteria primarily preserve large activations or reconstruct layer outputs. We argue that this overlooks a…
Understanding Empirical Unlearning with Combinatorial Interpretability
Shingo Kodama, Niv Cohen, Micah Adler +1
While many recent methods aim to unlearn or remove knowledge from pretrained models, seemingly erased knowledge often persists and can be recovered in various ways. Because large f…
Expand Neurons, Not Parameters
Linghao Kong, Inimai Subramanian, Yonadav Shavit +3
This work demonstrates how increasing the number of neurons in a network without increasing its total number of non-zero parameters improves performance. We show that this gain cor…
Negative Pre-activations Differentiate Syntax
Linghao Kong, Angelina Ning, Micah Adler +1
Modern large language models increasingly use smooth activation functions such as GELU or SiLU, allowing negative pre-activations to carry both signal and gradient. Nevertheless, m…
A Capacity-Based Rationale for Multi-Head Attention
Micah Adler
We study the capacity of the self-attention key-query channel: for a fixed budget, how many distinct token-token relations can a single layer reliably encode? We introduce Relation…
Towards Combinatorial Interpretability of Neural Computation
Micah Adler, Dan Alistarh, Nir Shavit
We introduce combinatorial interpretability, a methodology for understanding neural computation by analyzing the combinatorial structures in the sign-based categorization of a netw…