1 citations · 3 across the 7 of their papers we have counts for
6 papers · 1 filter
Batch-Paced: Fixed-Budget Batch Decoding in Discrete Diffusion Language Models
Shuai Zhang, Zhanchao Zhou, Huachuan Qiu +2
As generative language models become more capable and widely used, inference demand grows, making efficient serving increasingly important. Discrete diffusion language models (dLLM…
Every Token Counts: Generalizing 16M Ultra-Long Context in Large Language Models
Xiang Hu, Zhanchao Zhou, Ruiqi Liang +3
This work explores the challenge of building ``Machines that Can Remember'', framing long-term memory as the problem of efficient ultra-long context modeling. We argue that this re…
Knocking-Heads Attention
Zhanchao Zhou, Xiaodong Chen, Haoxing Chen +2
Multi-head attention (MHA) has become the cornerstone of modern large language models, enhancing representational capacity through parallel attention heads. However, increasing the…
Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts
Haoyuan Wu, Haoxing Chen, Xiaodong Chen +10
The Mixture of Experts (MoE) architecture is a cornerstone of modern state-of-the-art (SOTA) large language models (LLMs). MoE models facilitate scalability by enabling sparse para…
Value Residual Learning
Zhanchao Zhou, Tianyi Wu, Zhiyun Jiang +2
While Transformer models have achieved remarkable success in various domains, the effectiveness of information propagation through deep networks remains a critical challenge. Stand…
Dynamics of Instruction Fine-Tuning for Chinese Large Language Models
Chiyu Song, Zhanchao Zhou, Jianhao Yan +3
Instruction tuning is a burgeoning method to elicit the general intelligence of Large Language Models (LLMs). While numerous studies have examined the impact of factors such as dat…