Partial Answer of How Transformers Learn Automata
arXiv:2504.20395
Abstract
We introduce a novel framework for simulating finite automata using representation-theoretic semidirect products and Fourier modules, achieving more efficient Transformer-based implementations.