8 papers
The Sparsity Whisperer
Linghao Kong, Inimai Subramanian, Micah Adler +3
Pruning reduces the inference cost of large language models, but existing criteria primarily preserve large activations or reconstruct layer outputs. We argue that this overlooks a…
Expand Neurons, Not Parameters
Linghao Kong, Inimai Subramanian, Yonadav Shavit +3
This work demonstrates how increasing the number of neurons in a network without increasing its total number of non-zero parameters improves performance. We show that this gain cor…
Negative Pre-activations Differentiate Syntax
Linghao Kong, Angelina Ning, Micah Adler +1
Modern large language models increasingly use smooth activation functions such as GELU or SiLU, allowing negative pre-activations to carry both signal and gradient. Nevertheless, m…
On the Complexity of Neural Computation in Superposition
Micah Adler, Nir Shavit
Superposition, the ability of neural networks to represent more features than neurons, is increasingly seen as key to the efficiency of large models. This paper investigates the th…
Understanding Empirical Unlearning with Combinatorial Interpretability
Shingo Kodama, Niv Cohen, Micah Adler +1
While many recent methods aim to unlearn or remove knowledge from pretrained models, seemingly erased knowledge often persists and can be recovered in various ways. Because large f…
A Capacity-Based Rationale for Multi-Head Attention
Micah Adler
We study the capacity of the self-attention key-query channel: for a fixed budget, how many distinct token-token relations can a single layer reliably encode? We introduce Relation…