14 citations · 16 across the 9 of their papers we have counts for
6 papers · 1 filter
DLLM Agent: See Farther, Run Faster
Huiling Zhen, Weizhe Lin, Renxi Liu +15
Diffusion large language models (DLLMs) have emerged as an alternative to autoregressive (AR) decoding with appealing efficiency and modeling properties, yet their implications for…
Top 10 Open Challenges Steering the Future of Diffusion Language Model and Its Variants
Yunhe Wang, Kai Han, Huiling Zhen +13
The paradigm of Large Language Models (LLMs) is currently defined by auto-regressive (AR) architectures, which generate text through a sequential ``brick-by-brick'' process. Despit…
Nexus: Higher-Order Attention Mechanisms in Transformers
Hanting Chen, Chong Zhu, Kai Han +6
Transformers have achieved significant success across various domains, relying on self-attention to capture dependencies. However, the standard first-order attention mechanism is o…
Pangu Embedded: An Efficient Dual-system LLM Reasoner with Metacognition
Hanting Chen, Yasheng Wang, Kai Han +21
This work presents Pangu Embedded, an efficient Large Language Model (LLM) reasoner developed on Ascend Neural Processing Units (NPUs), featuring flexible fast and slow thinking ca…
Pangu Pro MoE: Mixture of Grouped Experts for Efficient Sparsity
Yehui Tang, Xiaosong Li, Fangcheng Liu +19
The surgence of Mixture of Experts (MoE) in Large Language Models promises a small price of execution cost for a much larger model parameter count and learning capacity, because on…
PanGu-: Enhancing Language Model Architectures via Nonlinearity Compensation
Yunhe Wang, Hanting Chen, Yehui Tang +17
The recent trend of large language models (LLMs) is to increase the scale of both model size (\aka the number of parameters) and dataset to achieve better generative ability, which…