Theory of Scaling Laws for In-Context Regression: Depth, Width, Context and Time

Blake Bordelon, Mary I. Letey, Cengiz Pehlevan

We study in-context learning (ICL) of linear regression in a deep linear self-attention model, characterizing how performance depends on various computational and statistical resou…