4 papers
Activation Outliers Matter: Robust Recovery for Quantized Multimodal LLMs
Tanzila Rahman, Mehran Taghian Jazi, Yunke Peng +10
Low-bit quantization offers a promising avenue for reducing the computational and memory demands of Multimodal Large Language Models (MLLMs). Recent hardware support for low-precis…
GNMR: Runtime Stability Control for Low-Precision Large Language Model Training
Boao Kong, Weichen Jia, Engao Zhang +6
Training stability is a key bottleneck in low-precision language model training: efficient low-cost paths can still produce short-lived numerical risks at a small set of operators.…
Pangu Ultra MoE: How to Train Your Big MoE on Ascend NPUs
Yehui Tang, Yichun Yin, Yaoyuan Wang +71
Sparse large language models (LLMs) with Mixture of Experts (MoE) and close to a trillion parameters are dominating the realm of most capable language models. However, the massive…
Pangu Ultra: Pushing the Limits of Dense Large Language Models on Ascend NPUs
Yichun Yin, Wenyong Huang, Kaikai Song +49
We present Pangu Ultra, a Large Language Model (LLM) with 135 billion parameters and dense Transformer modules trained on Ascend Neural Processing Units (NPUs). Although the field…