5 papers · 1 filter
ChunkLLM: A Lightweight Pluggable Framework for Accelerating LLMs Inference
Haojie Ouyang, Jianwei Lv, Lei Ren +3
Transformer-based large models excel in natural language processing and computer vision, but face severe computational inefficiencies due to the self-attention's quadratic complexi…
Lngram: N-gram Conditional Memory in Latent Space
Yunao Zheng, Guoyang Xia, Xiaojie Wang +1
Sequence modeling requires both compositional reasoning and local static knowledge retrieval, yet standard Transformers handle both through dense computation. Engram partially deco…
ROSA-Tuning: Enhancing Long-Context Modeling via Suffix Matching
Yunao Zheng, Xiaojie Wang, Lei Ren +1
Long-context capability and computational efficiency are among the central challenges facing today's large language models. Existing efficient attention methods reduce computationa…
Collab-Overcooked: Benchmarking and Evaluating Large Language Models as Collaborative Agents
Haochen Sun, Shuwen Zhang, Lujie Niu +6
Large Language Models (LLMs) based agent systems have made great strides in real-world applications beyond traditional NLP tasks. This paper proposes a new LLM-based Multi-Agent Sy…
SMAR: Soft Modality-Aware Routing Strategy for MoE-based Multimodal Large Language Models Preserving Language Capabilities
Guoyang Xia, Yifeng Ding, Fengfa Li +4
Mixture of Experts (MoE) architectures have become a key approach for scaling large language models, with growing interest in extending them to multimodal tasks. Existing methods t…