Showing 2024Show all
3 papers · 1 filter
cs.LG2024
Ultra-Sparse Memory Network
Zihao Huang, Qiyang Min, Hongzhi Huang +4
It is widely acknowledged that the performance of Transformer models is logarithmically related to their number of parameters and computational complexity. While approaches like Mi…
cs.CL2024
Polynomial Composition Activations: Unleashing the Dynamics of Large Language Models
Zhijian Zhuo, Ya Wang, Yutao Zeng +3
Transformers have found extensive applications across various domains due to the powerful fitting capabilities. This success can be partially attributed to their inherent nonlinear…
cs.LG2024
Hyper-Connections
Defa Zhu, Hongzhi Huang, Zihao Huang +5
We present hyper-connections, a simple yet effective method that can serve as an alternative to residual connections. This approach specifically addresses common drawbacks observed…