Multi-Gate Residuals

Zhizhan Zheng, Feiyun Zhang, Shuchun Liu +4

While Attention Residuals has shown some effectiveness in addressing the widespread issue of unbounded activation growth across deep residual layers, it inevitably incurs significa…