Protoreasoning in Tiny Transformers

Eduardo Valle, Fergal Reid

We show that tiny transformers can profitably employ a simple form of Chain of Thought, which we call protoreasoning, allowing us to study step-by-step reasoning on ~1M-parameter m…