Attention Is Not All You Need: The Importance of Feedforward Networks in Transformer Models

Isaac Gerber

Decoder-only transformer networks have become incredibly popular for language modeling tasks. State-of-the-art models can have over a hundred transformer blocks, containing billion…