TriPLU: Bypassing the Gate with Direct Trilinear Product FFNs in Tiny Language Models

He Zhang

We study whether tiny decoder-only language models benefit from feed-forward layers that directly multiply learned feature projections. TriPLU, a Trilinear Product Linear Unit, rep…