Bayesian Optimality of In-Context Learning with Selective State Spaces

Di Zhang, Jiaqi Xing

We propose Bayesian optimal sequential prediction as a new principle for understanding in-context learning (ICL). Unlike interpretations framing Transformers as performing implicit…