8 papers
Native Hybrid Attention for Efficient Sequence Modeling
Jusen Du, Jiaxi Hu, Tao Zhang +2
Transformers excel at sequence modeling but face quadratic complexity, while linear attention offers improved efficiency but often compromises recall accuracy over long contexts. I…
Nirvana: A Specialized Generalist Model With Task-Aware Memory Mechanism
Yuhua Jiang, Shuang Cheng, Yihao Liu +7
Large Language Models (LLMs) excel at general language tasks but struggle in specialized domains. Specialized Generalist Models (SGMs) address this by preserving broad capabilities…
A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyond
Xiaoye Qu, Yafu Li, Zhao-Chen Su +15
Recent Large Reasoning Models (LRMs), such as DeepSeek-R1 and OpenAI o1, have demonstrated strong performance gains by scaling up the length of Chain-of-Thought (CoT) reasoning dur…
Comba: Improving Bilinear RNNs with Closed-loop Control
Jiaxi Hu, Yongqi Pan, Jusen Du +5
Recent efficient sequence modeling methods such as Gated DeltaNet, TTT, and RWKV-7 have achieved performance improvements by supervising the recurrent memory management through Del…
MoM: Linear Sequence Modeling with Mixture-of-Memories
Jusen Du, Weigao Sun, Disen Lan +2
Linear sequence modeling methods, such as linear attention, state space modeling, and linear RNNs, offer significant efficiency improvements by reducing the complexity of training…
Speed Always Wins: A Survey on Efficient Architectures for Large Language Models
Weigao Sun, Jiaxi Hu, Yucheng Zhou +12
Large Language Models (LLMs) have delivered impressive results in language understanding, generation, reasoning, and pushes the ability boundary of multimodal models. Transformer m…