72 citations · 178 across the 28 of their papers we have counts for
30 papers
Lost in Tokenization: Fundamental Trade-offs in Graph Tokenization for Transformers
Maya Bechler-Speicher, Gilad Yehudai, Gil Harari +3
Transformers have become a central architecture for graph learning, but their application to graphs requires first choosing a tokenization: a graph-to-token map that determines whi…
Geometric Factual Recall in Transformers
Shauli Ravfogel, Gilad Yehudai, Joan Bruna +1
How do transformer language models memorize factual associations? A common view casts internal weight matrices as associative memories over pairs of embeddings, requiring parameter…
Emergence of Linear Truth Encodings in Language Models
Shauli Ravfogel, Gilad Yehudai, Tal Linzen +2
Recent probing studies reveal that large language models exhibit linear subspaces that separate true from false statements, yet the mechanism behind their emergence is unclear. We…
Provable Unlearning with Gradient Ascent on Two-Layer ReLU Neural Networks
Odelia Melamed, Gilad Yehudai, Gal Vardi
Machine Unlearning aims to remove specific data from trained models, addressing growing privacy and ethical concerns. We provide a theoretical analysis of a simple and widely used…
Querying Kernel Methods Suffices for Reconstructing their Training Data
Daniel Barzilai, Yuval Margalit, Eitan Gronich +3
Over-parameterized models have raised concerns about their potential to memorize training data, even when achieving strong generalization. The privacy implications of such memoriza…
Depth-Width tradeoffs in Algorithmic Reasoning of Graph Tasks with Transformers
Gilad Yehudai, Clayton Sanford, Maya Bechler-Speicher +3
Transformers have revolutionized the field of machine learning. In particular, they can be used to solve complex algorithmic problems, including graph-based tasks. In such algorith…
Compositional Reasoning with Transformers, RNNs, and Chain of Thought
Gilad Yehudai, Noah Amsel, Joan Bruna
It is well understood that different neural network architectures are suited to different tasks, but is there always a single best architecture for a given task? We compare the exp…
Logarithmic Width Suffices for Robust Memorization
Amitsour Egosi, Gilad Yehudai, Ohad Shamir
The memorization capacity of neural networks with a given architecture has been thoroughly studied in many works. Specifically, it is well-known that memorizing samples can be…
On the Reconstruction of Training Data from Group Invariant Networks
Ran Elbaz, Gilad Yehudai, Meirav Galun +1
Reconstructing training data from trained neural networks is an active area of research with significant implications for privacy and explainability. Recent advances have demonstra…
On the Benefits of Rank in Attention Layers
Noah Amsel, Gilad Yehudai, Joan Bruna
Attention-based mechanisms are widely used in machine learning, most prominently in transformers. However, hyperparameters such as the rank of the attention matrices and the number…
Reconstructing Training Data From Real World Models Trained with Transfer Learning
Yakir Oz, Gilad Yehudai, Gal Vardi +3
Current methods for reconstructing training data from trained classifiers are restricted to very small models, limited training set sizes, and low-resolution images. Such restricti…
MALT Powers Up Adversarial Attacks
Odelia Melamed, Gilad Yehudai, Adi Shamir
Current adversarial attacks for multi-class classifiers choose the target class for a given input naively, based on the classifier's confidence levels for various target classes. W…
When Can Transformers Count to n?
Gilad Yehudai, Haim Kaplan, Guy Dar +4
Large language models based on the transformer architecture can solve highly complex tasks, yet their fundamental limitations on simple algorithmic problems remain poorly understoo…
RedEx: Beyond Fixed Representation Methods via Convex Optimization
Amit Daniely, Mariano Schain, Gilad Yehudai
Optimizing Neural networks is a difficult task which is still not well understood. On the other hand, fixed representation methods such as kernels and random features have provable…
Locally Optimal Descent for Dynamic Stepsize Scheduling
Gilad Yehudai, Alon Cohen, Amit Daniely +3
We introduce a novel dynamic learning-rate scheduling scheme grounded in theory with the goal of simplifying the manual and time-consuming tuning of schedules in practice. Our appr…
Deconstructing Data Reconstruction: Multiclass, Weight Decay and General Losses
Gon Buzaglo, Niv Haim, Gilad Yehudai +4
Memorization of training data is an active research area, yet our understanding of the inner workings of neural networks is still in its infancy. Recently, Haim et al. (2022) propo…
Reconstructing Training Data from Multiclass Neural Networks
Gon Buzaglo, Niv Haim, Gilad Yehudai +2
Reconstructing samples from the training set of trained neural networks is a major privacy concern. Haim et al. (2022) recently showed that it is possible to reconstruct training s…
From Tempered to Benign Overfitting in ReLU Neural Networks
Guy Kornowski, Gilad Yehudai, Ohad Shamir
Overparameterized neural networks (NNs) are observed to generalize well even when trained to perfectly fit noisy data. This phenomenon motivated a large body of work on "benign ove…
Adversarial Examples Exist in Two-Layer ReLU Networks for Low Dimensional Linear Subspaces
Odelia Melamed, Gilad Yehudai, Gal Vardi
Despite a great deal of research, it is still not well-understood why trained neural networks are highly vulnerable to adversarial examples. In this work we focus on two-layer neur…
Reconstructing Training Data from Trained Neural Networks
Niv Haim, Gal Vardi, Gilad Yehudai +2
Understanding to what extent neural networks memorize training data is an intriguing question with practical and theoretical implications. In this paper we show that in some cases…
Gradient Methods Provably Converge to Non-Robust Networks
Gal Vardi, Gilad Yehudai, Ohad Shamir
Despite a great deal of research, it is still unclear why neural networks are so susceptible to adversarial examples. In this work, we identify natural settings where depth- ReL…
Width is Less Important than Depth in ReLU Neural Networks
Gal Vardi, Gilad Yehudai, Ohad Shamir
We solve an open question from Lu et al. (2017), by showing that any target network with inputs in can be approximated by a width network (independent of the…
On the Optimal Memorization Power of ReLU Neural Networks
Gal Vardi, Gilad Yehudai, Ohad Shamir
We study the memorization power of feedforward ReLU neural networks. We show that such networks can memorize any points that satisfy a mild separability assumption using $\tild…
Learning a Single Neuron with Bias Using Gradient Descent
Gal Vardi, Gilad Yehudai, Ohad Shamir
We theoretically study the fundamental problem of learning a single neuron with a bias term () in the realizable setting with the…