Single layer tiny Co${}^4$ outpaces GPT-2 and GPT-BERT

Noor Ul Zain, Mohsin Raza, Ahsan Adeel

We show that a tiny Co${}^4$ machine(Adeel,2025) with a single layer, two heads, and 8M parameters, operating at an approximate cost of $O(N)$ (where $N$ is the number of input token…