Masked Gated Linear Unit

Yukito Tajima, Nakamasa Inoue, Yusuke Sekikawa +2

Gated Linear Units (GLUs) have become essential components in the feed-forward networks of state-of-the-art Large Language Models (LLMs). However, they require twice as many memory…