49 citations · 151 across the 32 of their papers we have counts for
27 papers · 1 filter
Which Tasks Survive Self-Supervised Learning?
Achleshwar Luthra, Lucas Bryant, Tracy Zhu +1
Same-instance self-supervised learning (SSL) learns representations by enforcing consistency across two views of the same underlying instance. This principle alone, however, does n…
Directional Neural Collapse Explains Few-Shot Transfer in Self-Supervised Learning
Achleshwar Luthra, Yash Salunkhe, Tomer Galanti
Frozen self-supervised representations often transfer well with only a few labels across many semantic tasks. We argue that a single geometric quantity, \emph{directional} CDNV (de…
Scalable Principal-Agent Contract Design via Gradient-Based Optimization
Tomer Galanti, Aarya Bookseller, Korok Ray
We study a bilevel \emph{max-max} optimization framework for principal-agent contract design, in which a principal chooses incentives to maximize utility while anticipating the age…
On the Alignment Between Supervised and Self-Supervised Contrastive Learning
Achleshwar Luthra, Priyadarsi Mishra, Tomer Galanti
Self-supervised contrastive learning (CL) has achieved remarkable empirical success, often producing representations that rival supervised pre-training on downstream tasks. Recent…
LLM Priors for ERM over Programs
Shivam Singhal, Priyadarsi Mishra, Eran Malach +1
We study program-learning methods that are efficient in both samples and computation. Classical learning theory suggests that when the target admits a short program description, fo…
Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learning
Achleshwar Luthra, Tianbao Yang, Tomer Galanti
Despite its empirical success, the theoretical foundations of self-supervised contrastive learning (CL) are not yet fully established. In this work, we address this gap by showing…
DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimization
Gang Li, Ming Lin, Tomer Galanti +2
The recent success and openness of DeepSeek-R1 have brought widespread attention to Group Relative Policy Optimization (GRPO) as a reinforcement learning method for large reasoning…
Formation of Representations in Neural Networks
Liu Ziyin, Isaac Chuang, Tomer Galanti +1
Understanding neural representations will help open the black box of neural networks and advance our scientific understanding of modern AI systems. However, how complex, structured…
Centered Self-Attention Layers
Ameen Ali, Tomer Galanti, Lior Wolf
The self-attention mechanism in transformers and the message-passing mechanism in graph neural networks are repeatedly applied within deep learning architectures. We show that this…
Reverse Engineering Self-Supervised Learning
Ido Ben-Shaul, Ravid Shwartz-Ziv, Tomer Galanti +2
Self-supervised learning (SSL) is a powerful tool in machine learning, but understanding the learned representations and their underlying mechanisms remains a challenge. This paper…
Type-II Saddles and Probabilistic Stability of Stochastic Gradient Descent
Liu Ziyin, Botao Li, Tomer Galanti +1
Characterizing and understanding the dynamics of stochastic gradient descent (SGD) around saddle points remains an open problem. We first show that saddle points in neural networks…
Norm-based Generalization Bounds for Compositionally Sparse Neural Networks
Tomer Galanti, Mengjia Xu, Liane Galanti +1
In this paper, we investigate the Rademacher complexity of deep sparse neural networks, where each neuron receives a small number of inputs. We prove generalization bounds for mult…
Exploring the Approximation Capabilities of Multiplicative Neural Networks for Smooth Functions
Ido Ben-Shaul, Tomer Galanti, Shai Dekel
Multiplication layers are a key component in various influential neural network modules, including self-attention and hypernetwork layers. In this paper, we investigate the approxi…
Generalization Bounds for Few-Shot Transfer Learning with Pretrained Classifiers
Tomer Galanti, András György, Marcus Hutter
We study the ability of foundation models to learn representations for classification that are transferable to new, unseen classes. Recent results in the literature show that repre…
SGD and Weight Decay Secretly Minimize the Rank of Your Neural Network
Tomer Galanti, Zachary S. Siegel, Aparna Gupte +1
We investigate the inherent bias of Stochastic Gradient Descent (SGD) toward learning low-rank weight matrices during the training of deep neural networks. Our results demonstrate…
On the Implicit Bias Towards Minimal Depth of Deep Neural Networks
Tomer Galanti, Liane Galanti, Ido Ben-Shaul
Recent results in the literature suggest that the penultimate (second-to-last) layer representations of neural networks that are trained for classification exhibit a clustering pro…
On the Role of Neural Collapse in Transfer Learning
Tomer Galanti, András György, Marcus Hutter
We study the ability of foundation models to learn representations for classification that are transferable to new, unseen classes. Recent results in the literature show that repre…
Weakly Supervised Recovery of Semantic Attributes
Ameen Ali, Tomer Galanti, Evgeniy Zheltonozhskiy +2
We consider the problem of the extraction of semantic attributes, supervised only with classification labels. For example, when learning to classify images of birds into species, w…
On Infinite-Width Hypernetworks
Etai Littwin, Tomer Galanti, Lior Wolf +1
{\em Hypernetworks} are architectures that produce the weights of a task-specific {\em primary network}. A notable application of hypernetworks in the recent literature involves le…
A Critical View of the Structural Causal Model
Tomer Galanti, Ofir Nabati, Lior Wolf
In the univariate case, we show that by comparing the individual complexities of univariate cause and effect, one can identify the cause and the effect, without considering their i…
On the Modularity of Hypernetworks
Tomer Galanti, Lior Wolf
In the context of learning to map an input to a function , two alternative methods are compared: (i) an embedding-based method, which learns a fi…
A Formal Approach to Explainability
Lior Wolf, Tomer Galanti, Tamir Hazan
We regard explanations as a blending of the input sample and the model's output and offer a few definitions that capture various desired properties of the function that generates t…
Unsupervised Learning of the Set of Local Maxima
Lior Wolf, Sagie Benaim, Tomer Galanti
This paper describes a new form of unsupervised learning, whose input is a set of unlabeled points that are assumed to be local maxima of an unknown value function v in an unknown…
Risk Bounds for Unsupervised Cross-Domain Mapping with IPMs
Tomer Galanti, Sagie Benaim, Lior Wolf
The recent empirical success of unsupervised cross-domain mapping algorithms, between two domains that share common characteristics, is not well-supported by theoretical justificat…