Attention-Only Transformers and Implementing MLPs with Attention Heads

Robert Huben, Valerie Morris

The transformer architecture is widely used in machine learning models and consists of two alternating sublayers: attention heads and MLPs. We prove that an MLP neuron can be imple…