Hybrid Gated Attention

Zekun Zhou, Ruobing Xie, Lanrui Wang +1

Gated attention is an effective approach to mitigate attention sinks and enhance the representational capacity of attention. To further extend its effectiveness-efficiency Pareto f…