Algorithmic Capabilities of Random Transformers

Ziqian Zhong, Jacob Andreas

Trained transformer models have been found to implement interpretable procedures for tasks like arithmetic and associative recall, but little is understood about how the circuits t…