3 papers
cs.LG2026
Confidence-Adaptive SwiGLU for Mixture-of-Experts
Shaohua Li, Xiuchao Sui, Xiaobing Sun +4
SwiGLU has become a standard gated activation in modern Transformer MLPs, yet its gate sharpness -- the smoothness and selectivity of the gating function -- is typically fixed thro…
cs.CL2023
Unraveling Feature Extraction Mechanisms in Neural Networks
Xiaobing Sun, Jiaxi Li, Wei Lu
The underlying mechanism of neural networks in capturing precise knowledge has been the subject of consistent research efforts. In this work, we propose a theoretical approach base…
cs.CL2022
Implicit N-grams Induced by Recurrence
Xiaobing Sun, Wei Lu
Although self-attention based models such as Transformers have achieved remarkable successes on natural language processing (NLP) tasks, recent studies reveal that they have limita…