5 papers
GRR-CoCa: Leveraging LLM Mechanisms in Multimodal Model Architectures
Jake R. Patock, Nicole Catherine Lewis, Kevin McCoy +3
State-of-the-art (SOTA) image and text generation models are multimodal models that have many similarities to large language models (LLMs). Despite achieving strong performances, l…
Pangu Pro MoE: Mixture of Grouped Experts for Efficient Sparsity
Yehui Tang, Xiaosong Li, Fangcheng Liu +19
The surgence of Mixture of Experts (MoE) in Large Language Models promises a small price of execution cost for a much larger model parameter count and learning capacity, because on…
Pangu Light: Weight Re-Initialization for Pruning and Accelerating LLMs
Hanting Chen, Jiarui Qin, Jialong Guo +15
Large Language Models (LLMs) deliver state-of-the-art capabilities across numerous tasks, but their immense size and inference costs pose significant computational challenges for p…
Pangu Ultra MoE: How to Train Your Big MoE on Ascend NPUs
Yehui Tang, Yichun Yin, Yaoyuan Wang +71
Sparse large language models (LLMs) with Mixture of Experts (MoE) and close to a trillion parameters are dominating the realm of most capable language models. However, the massive…
Pangu Ultra: Pushing the Limits of Dense Large Language Models on Ascend NPUs
Yichun Yin, Wenyong Huang, Kaikai Song +49
We present Pangu Ultra, a Large Language Model (LLM) with 135 billion parameters and dense Transformer modules trained on Ascend Neural Processing Units (NPUs). Although the field…