most citedIn-Context Learning of a Linear Transformer Block: Benefits of the MLP Component and One-Step GD Initialization

1 citations · 1 across the 2 of their papers we have counts for

collaborators

2 papers

stat.ML2024

Large Stepsize Gradient Descent for Non-Homogeneous Two-Layer Networks: Margin Improvement and Fast Optimization

Yuhang Cai, Jingfeng Wu, Song Mei +2

The typical training of neural networks using large stepsize gradient descent (GD) under the logistic loss often involves two distinct phases, where the empirical risk oscillates i…

stat.ML2024★ 1 cited

In-Context Learning of a Linear Transformer Block: Benefits of the MLP Component and One-Step GD Initialization

Ruiqi Zhang, Jingfeng Wu, Peter L. Bartlett

We study the \emph{in-context learning} (ICL) ability of a \emph{Linear Transformer Block} (LTB) that combines a linear attention component and a linear multi-layer perceptron (MLP…