Weight-based Decomposition: A Case for Bilinear MLPs

Michael T. Pearce, Thomas Dooms, Alice Rigg

Gated Linear Units (GLUs) have become a common building block in modern foundation models. Bilinear layers drop the non-linearity in the "gate" but still have comparable performanc…