4 papers · 1 filter
Understanding In-Context Learning of Linear Models in Transformers Through an Adversarial Lens
Usman Anwar, Johannes Von Oswald, Louis Kirsch +2
In this work, we make two contributions towards understanding of in-context learning of linear models by transformers. First, we investigate the adversarial robustness of in-contex…
Benign Overfitting in Single-Head Attention
Roey Magen, Shuning Shang, Zhiwei Xu +3
The phenomenon of benign overfitting, where a trained neural network perfectly fits noisy training data but still achieves near-optimal test performance, has been extensively studi…
Trained Transformer Classifiers Generalize and Exhibit Benign Overfitting In-Context
Spencer Frei, Gal Vardi
Transformers have the capacity to act as supervised learning algorithms: by properly encoding a set of labeled training ("in-context") examples and an unlabeled test example into a…
Minimum-Norm Interpolation Under Covariate Shift
Neil Mallinar, Austin Zane, Spencer Frei +1
Transfer learning is a critical part of real-world machine learning deployments and has been extensively studied in experimental works with overparameterized neural networks. Howev…