49 citations · 151 across the 32 of their papers we have counts for
38 papers
Embedding Geometry Shapes LLM Routing: Tracing Failure Modes and Improving Generalization
Leonardo Torti, Emanuele Rimoldi, Tomer Galanti +2
With the growing number of LLMs available, selecting which one to use for a prompt has become a key question. A large part of the routing literature relies on semantically embeddin…
A Four-by-Four Obstruction to Polynomial Removal for Ordered Binary Matrices
Zinan Wang
We exhibit a fixed binary matrix with no polynomial induced ordered removal bound. At depth , let . Our explicit host has order , exactly $2m^…
Which Tasks Survive Self-Supervised Learning?
Achleshwar Luthra, Lucas Bryant, Tracy Zhu +1
Same-instance self-supervised learning (SSL) learns representations by enforcing consistency across two views of the same underlying instance. This principle alone, however, does n…
Distribution-Aware Programming: Learning Specialized Solvers from Experience
Saharsh Koganti, Priyadarsi Mishra, Pierfrancesco Beneventano +1
Many optimization problems are solved repeatedly on instances drawn from the same underlying distribution. In this setting, a system can begin with a general-purpose solver and use…
Inference-Time Amplification of Weak Reasoning Models
Varun Sunkaraneni, Pierfrancesco Beneventano, Riccardo Neumarker +2
How much of the capability of a reasoning model exposed by repeated sampling can an imperfect selector recover? We study this problem as {\em inference-time amplification}, separat…
Directional Neural Collapse Explains Few-Shot Transfer in Self-Supervised Learning
Achleshwar Luthra, Yash Salunkhe, Tomer Galanti
Frozen self-supervised representations often transfer well with only a few labels across many semantic tasks. We argue that a single geometric quantity, \emph{directional} CDNV (de…
Scalable Principal-Agent Contract Design via Gradient-Based Optimization
Tomer Galanti, Aarya Bookseller, Korok Ray
We study a bilevel \emph{max-max} optimization framework for principal-agent contract design, in which a principal chooses incentives to maximize utility while anticipating the age…
On the Alignment Between Supervised and Self-Supervised Contrastive Learning
Achleshwar Luthra, Priyadarsi Mishra, Tomer Galanti
Self-supervised contrastive learning (CL) has achieved remarkable empirical success, often producing representations that rival supervised pre-training on downstream tasks. Recent…
LLM Priors for ERM over Programs
Shivam Singhal, Priyadarsi Mishra, Eran Malach +1
We study program-learning methods that are efficient in both samples and computation. Classical learning theory suggests that when the target admits a short program description, fo…
Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learning
Achleshwar Luthra, Tianbao Yang, Tomer Galanti
Despite its empirical success, the theoretical foundations of self-supervised contrastive learning (CL) are not yet fully established. In this work, we address this gap by showing…
DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimization
Gang Li, Ming Lin, Tomer Galanti +2
The recent success and openness of DeepSeek-R1 have brought widespread attention to Group Relative Policy Optimization (GRPO) as a reinforcement learning method for large reasoning…
The Fair Language Model Paradox
Andrea Pinto, Tomer Galanti, Randall Balestriero
Large Language Models (LLMs) are widely deployed in real-world applications, yet little is known about their training dynamics at the token level. Evaluation typically relies on ag…
Formation of Representations in Neural Networks
Liu Ziyin, Isaac Chuang, Tomer Galanti +1
Understanding neural representations will help open the black box of neural networks and advance our scientific understanding of modern AI systems. However, how complex, structured…
On the Power of Decision Trees in Auto-Regressive Language Modeling
Yulu Gan, Tomer Galanti, Tomaso Poggio +1
Originally proposed for handling time series data, Auto-regressive Decision Trees (ARDTs) have not yet been explored for language modeling. This paper delves into both the theoreti…
Distributed Speculative Inference (DSI): Speculation Parallelism for Provably Faster Lossless Language Model Inference
Nadav Timor, Jonathan Mamou, Daniel Korat +6
This paper introduces distributed speculative inference (DSI), a novel inference algorithm that is provably faster than speculative inference (SI) [leviathan2023, chen2023, miao202…
Centered Self-Attention Layers
Ameen Ali, Tomer Galanti, Lior Wolf
The self-attention mechanism in transformers and the message-passing mechanism in graph neural networks are repeatedly applied within deep learning architectures. We show that this…
Reverse Engineering Self-Supervised Learning
Ido Ben-Shaul, Ravid Shwartz-Ziv, Tomer Galanti +2
Self-supervised learning (SSL) is a powerful tool in machine learning, but understanding the learned representations and their underlying mechanisms remains a challenge. This paper…
Type-II Saddles and Probabilistic Stability of Stochastic Gradient Descent
Liu Ziyin, Botao Li, Tomer Galanti +1
Characterizing and understanding the dynamics of stochastic gradient descent (SGD) around saddle points remains an open problem. We first show that saddle points in neural networks…
Norm-based Generalization Bounds for Compositionally Sparse Neural Networks
Tomer Galanti, Mengjia Xu, Liane Galanti +1
In this paper, we investigate the Rademacher complexity of deep sparse neural networks, where each neuron receives a small number of inputs. We prove generalization bounds for mult…
Exploring the Approximation Capabilities of Multiplicative Neural Networks for Smooth Functions
Ido Ben-Shaul, Tomer Galanti, Shai Dekel
Multiplication layers are a key component in various influential neural network modules, including self-attention and hypernetwork layers. In this paper, we investigate the approxi…
Generalization Bounds for Few-Shot Transfer Learning with Pretrained Classifiers
Tomer Galanti, András György, Marcus Hutter
We study the ability of foundation models to learn representations for classification that are transferable to new, unseen classes. Recent results in the literature show that repre…
SGD and Weight Decay Secretly Minimize the Rank of Your Neural Network
Tomer Galanti, Zachary S. Siegel, Aparna Gupte +1
We investigate the inherent bias of Stochastic Gradient Descent (SGD) toward learning low-rank weight matrices during the training of deep neural networks. Our results demonstrate…
On the Implicit Bias Towards Minimal Depth of Deep Neural Networks
Tomer Galanti, Liane Galanti, Ido Ben-Shaul
Recent results in the literature suggest that the penultimate (second-to-last) layer representations of neural networks that are trained for classification exhibit a clustering pro…
On the Role of Neural Collapse in Transfer Learning
Tomer Galanti, András György, Marcus Hutter
We study the ability of foundation models to learn representations for classification that are transferable to new, unseen classes. Recent results in the literature show that repre…