373 citations · 1k across the 52 of their papers we have counts for
8 papers · 1 filter
Linearized Relative Positional Encoding
Zhen Qin, Weixuan Sun, Kaiyue Lu +6
Relative positional encoding is widely used in vanilla and linear transformers to represent positional information. However, existing encoding methods of a vanilla transformer are…
TransNormerLLM: A Faster and Better Large Language Model with Improved TransNormer
Zhen Qin, Dong Li, Weigao Sun +8
We present TransNormerLLM, the first linear attention-based Large Language Model (LLM) that outperforms conventional softmax attention-based models in terms of both accuracy and ef…
Exploring Transformer Extrapolation
Zhen Qin, Yiran Zhong, Hui Deng
Length extrapolation has attracted considerable attention recently since it allows transformers to be tested on longer sequences than those used in training. Previous research has…
Toeplitz Neural Network for Sequence Modeling
Zhen Qin, Xiaodong Han, Weixuan Sun +6
Sequence modeling has important applications in natural language processing and computer vision. Recently, the transformer-based models have shown strong performance on various seq…
The Devil in Linear Transformer
Zhen Qin, XiaoDong Han, Weixuan Sun +4
Linear transformers aim to reduce the quadratic space-time complexity of vanilla transformers. However, they usually suffer from degraded performances on various tasks and corpus.…
Neural Architecture Search on Efficient Transformers and Beyond
Zexiang Liu, Dong Li, Kaiyue Lu +4
Recently, numerous efficient Transformers have been proposed to reduce the quadratic computational complexity of standard Transformers caused by the Softmax attention. However, mos…
cosFormer: Rethinking Softmax in Attention
Zhen Qin, Weixuan Sun, Hui Deng +6
Transformer has shown great successes in natural language processing, computer vision, and audio processing. As one of its core components, the softmax attention helps to capture l…
Transcribing Natural Languages for The Deaf via Neural Editing Programs
Dongxu Li, Chenchen Xu, Liu Liu +4
This work studies the task of glossification, of which the aim is to em transcribe natural spoken language sentences for the Deaf (hard-of-hearing) community to ordered sign langua…