34 citations · 76 across the 12 of their papers we have counts for
3 papers · 1 filter
On the Optimization and Generalization of Multi-head Attention
Puneesh Deora, Rouzbeh Ghaderi, Hossein Taheri +1
The training and generalization dynamics of the Transformer's core mechanism, namely the Attention mechanism, remain under-explored. Besides, existing analyses primarily focus on s…
Fast Convergence in Learning Two-Layer Neural Networks with Separable Data
Hossein Taheri, Christos Thrampoulidis
Normalized gradient descent has shown substantial success in speeding up the convergence of exponentially-tailed loss functions (which includes exponential and logistic losses) on…
Generalization and Stability of Interpolating Neural Networks with Minimal Width
Hossein Taheri, Christos Thrampoulidis
We investigate the generalization and optimization properties of shallow neural-network classifiers trained by gradient descent in the interpolating regime. Specifically, in a real…