1 citations · 1 across the 3 of their papers we have counts for
5 papers · 1 filter
Beyond Reward Engineering: A Data Recipe for Long-Context Reinforcement Learning
Xiaoyue Xu, Sikui Zhang, Xiaorong Wang +2
Long-context reasoning is an essential capability for large language models, particularly when they are deployed as autonomous agents that must reason over lengthy trajectories. Re…
Rethinking the Role of Efficient Attention in Hybrid Architectures
Ziqing Qiao, Yinuo Xu, Chaojun Xiao +6
Modern language models increasingly adopt hybrid architectures that combine full attention with efficient attention modules, such as sliding-window attention (SWA) and recurrent se…
MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modeling
MiniCPM Team, Wenhao An, Yingfa Chen +44
The evolution of large language models (LLMs) towards applications with ultra-long contexts faces challenges posed by the high computational and memory costs of the Transformer arc…
Stress-Testing Long-Context Language Models with Lifelong ICL and Task Haystack
Xiaoyue Xu, Qinyuan Ye, Xiang Ren
We introduce Lifelong ICL, a problem setting that challenges long-context language models (LMs) to learn a sequence of language tasks through in-context learning (ICL). We further…
Boosting Inference Efficiency: Unleashing the Power of Parameter-Shared Pre-trained Language Models
Weize Chen, Xiaoyue Xu, Xu Han +5
Parameter-shared pre-trained language models (PLMs) have emerged as a successful approach in resource-constrained environments, enabling substantial reductions in model storage and…