Tracr-Injection: Distilling Algorithms into Pre-trained Language Models

Tomás Vergara-Browne, Álvaro Soto

Motivated by the surge of large language models, there has been a push to formally characterize the symbolic abilities intrinsic to the transformer architecture. A programming lang…