2 citations · 3 across the 10 of their papers we have counts for
11 papers
Demystifying the Slash Pattern in Attention: The Role of RoPE
Yuan Cheng, Fengzhuo Zhang, Yunlong Hou +5
Large Language Models (LLMs) often exhibit slash attention patterns, where attention scores concentrate along the -th sub-diagonal for some offset . These patterns play a key…
Annealed Relaxation of Speculative Decoding for Faster Autoregressive Image Generation
Xingyao Li, Fengzhuo Zhang, Cunxiao Du +1
Despite significant progress in autoregressive image generation, inference remains slow due to the sequential nature of AR models and the ambiguity of image tokens, even when using…
Muon Outperforms Adam in Tail-End Associative Memory Learning
Shuche Wang, Fengzhuo Zhang, Jiaxiang Li +6
The Muon optimizer is consistently faster than Adam in training Large Language Models (LLMs), yet the mechanism underlying its success remains unclear. This paper demystifies this…
Sparse-to-Dense: A Free Lunch for Lossless Acceleration of Video Understanding in LLMs
Xuan Zhang, Cunxiao Du, Sicheng Yu +4
Due to the auto-regressive nature of current video large language models (Video-LLMs), the inference latency increases as the input sequence length grows, posing challenges for the…
BanditSpec: Adaptive Speculative Decoding via Bandit Algorithms
Yunlong Hou, Fengzhuo Zhang, Cunxiao Du +6
Speculative decoding has emerged as a popular method to accelerate the inference of Large Language Models (LLMs) while retaining their superior text generation performance. Previou…
Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMs
Longxu Dou, Qian Liu, Fan Zhou +38
Sailor2 is a family of cutting-edge multilingual language models for South-East Asian (SEA) languages, available in 1B, 8B, and 20B sizes to suit diverse applications. Building on…
LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verification
Penghui Yang, Cunxiao Du, Fengzhuo Zhang +4
As Large Language Models (LLMs) can now process extremely long contexts, efficient inference over these extended inputs has become increasingly important, especially for emerging a…
When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Training
Haonan Wang, Qian Liu, Chao Du +4
Extending context window sizes allows large language models (LLMs) to process longer sequences and handle more complex tasks. Rotary Positional Embedding (RoPE) has become the de f…
LightTransfer: Your Long-Context LLM is Secretly a Hybrid Model with Effortless Adaptation
Xuan Zhang, Fengzhuo Zhang, Cunxiao Du +4
Scaling language models to handle longer contexts introduces substantial memory challenges due to the growing cost of key-value (KV) caches. Motivated by the efficiency gains of hy…
When Attention Sink Emerges in Language Models: An Empirical View
Xiangming Gu, Tianyu Pang, Chao Du +5
Language Models (LMs) assign significant attention to the first token, even if it is not semantically important, which is known as attention sink. This phenomenon has been widely a…
Reverse Modeling in Large Language Models
Sicheng Yu, Yuanchen Xu, Cunxiao Du +5
Humans are accustomed to reading and writing in a forward manner, and this natural bias extends to text understanding in auto-regressive large language models (LLMs). This paper in…