11 citations · 14 across the 4 of their papers we have counts for
4 papers
Centered Self-Attention Layers
Ameen Ali, Tomer Galanti, Lior Wolf
The self-attention mechanism in transformers and the message-passing mechanism in graph neural networks are repeatedly applied within deep learning architectures. We show that this…
Reverse Engineering Self-Supervised Learning
Ido Ben-Shaul, Ravid Shwartz-Ziv, Tomer Galanti +2
Self-supervised learning (SSL) is a powerful tool in machine learning, but understanding the learned representations and their underlying mechanisms remains a challenge. This paper…
Norm-based Generalization Bounds for Compositionally Sparse Neural Networks
Tomer Galanti, Mengjia Xu, Liane Galanti +1
In this paper, we investigate the Rademacher complexity of deep sparse neural networks, where each neuron receives a small number of inputs. We prove generalization bounds for mult…
Exploring the Approximation Capabilities of Multiplicative Neural Networks for Smooth Functions
Ido Ben-Shaul, Tomer Galanti, Shai Dekel
Multiplication layers are a key component in various influential neural network modules, including self-attention and hypernetwork layers. In this paper, we investigate the approxi…