1 citations · 1 across the 4 of their papers we have counts for
6 papers
LLaDA2.0: Scaling Up Diffusion Language Models to 100B
Tiwei Bie, Maosong Cao, Kun Chen +28
This paper presents LLaDA2.0 -- a tuple of discrete diffusion large language models (dLLM) scaling up to 100B total parameters through systematic conversion from auto-regressive (A…
Every Token Counts: Generalizing 16M Ultra-Long Context in Large Language Models
Xiang Hu, Zhanchao Zhou, Ruiqi Liang +3
This work explores the challenge of building ``Machines that Can Remember'', framing long-term memory as the problem of efficient ultra-long context modeling. We argue that this re…
Knocking-Heads Attention
Zhanchao Zhou, Xiaodong Chen, Haoxing Chen +2
Multi-head attention (MHA) has become the cornerstone of modern large language models, enhancing representational capacity through parallel attention heads. However, increasing the…
Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts
Haoyuan Wu, Haoxing Chen, Xiaodong Chen +10
The Mixture of Experts (MoE) architecture is a cornerstone of modern state-of-the-art (SOTA) large language models (LLMs). MoE models facilitate scalability by enabling sparse para…
QUBE: Enhancing Automatic Heuristic Design via Quality-Uncertainty Balanced Evolution
Zijie Chen, Zhanchao Zhou, Yu Lu +3
Solving NP-hard problems traditionally relies on heuristics, yet manually designing effective heuristics for complex problems remains a significant challenge. While recent advancem…
Value Residual Learning
Zhanchao Zhou, Tianyi Wu, Zhiyun Jiang +2
While Transformer models have achieved remarkable success in various domains, the effectiveness of information propagation through deep networks remains a critical challenge. Stand…