Markov Chain Estimation with In-Context Learning

Simon Lepage, Jeremie Mary, David Picard

We investigate the capacity of transformers to learn algorithms involving their context while solely being trained using next token prediction. We set up Markov chains with random…