Superiority of Multi-Head Attention in In-Context Linear Regression

Yingqian Cui, Jie Ren, Pengfei He +2

We present a theoretical analysis of the performance of transformer with softmax attention in in-context learning with linear regression tasks. While the existing literature predom…