8 papers · 1 filter
Training Transformers for KV Cache Compressibility
Yoav Gelberg, Yam Eitan, Michael Bronstein +2
Long-context language modeling is increasingly constrained by the Key-Value (KV) cache, whose memory and decode-time access costs scale linearly with the prefix length. This bottle…
Muon is Not That Special: Random or Inverted Spectra Work Just as Well
Zakhar Shumaylov, Nathaël Da Costa, Peter Zaika +6
The recent empirical success of the Muon optimizer has renewed interest in non-Euclidean optimization, typically justified by similarities with second-order methods, and linear min…
On The Expressive Power of GNN Derivatives
Yam Eitan, Moshe Eliasof, Yoav Gelberg +3
Despite significant advances in Graph Neural Networks (GNNs), their limited expressivity remains a fundamental challenge. Research on GNN expressivity has produced many expressive…
GradMetaNet: An Equivariant Architecture for Learning on Gradients
Yoav Gelberg, Yam Eitan, Aviv Navon +5
Gradients of neural networks encode valuable information for optimization, editing, and analysis of models. Therefore, practitioners often treat gradients as inputs to task-specifi…
Beyond Next Token Probabilities: Learnable, Fast Detection of Hallucinations and Data Contamination on LLM Output Distributions
Guy Bar-Shalom, Fabrizio Frasca, Derek Lim +5
The automated detection of hallucinations and training data contamination is pivotal to the safe deployment of Large Language Models (LLMs). These tasks are particularly challengin…
Learning on LoRAs: GL-Equivariant Processing of Low-Rank Weight Spaces for Large Finetuned Models
Theo Putterman, Derek Lim, Yoav Gelberg +2
Low-rank adaptations (LoRAs) have revolutionized the finetuning of large foundation models, enabling efficient adaptation even with limited computational resources. The resulting p…