Publications (5)
CommonIT: Commonality-Aware Instruction Tuning for Large Language Models via Data Partitions
Jun Rao, Xuebo Liu, Lian Lian +3
With instruction tuning, Large Language Models (LLMs) can enhance their ability to adhere to commands. Diverging from most works focusing on data mixing, our study concentrates on…
SeaPO: Strategic Error Amplification for Robust Preference Optimization of Large Language Models
Jun Rao, Yunjie Liao, Xuebo Liu +6
Existing alignment methods for preference optimization of large language models (LLMs) aim to enhance model performance by utilizing pairs of positive and negative samples. However…
APT: Improving Specialist LLM Performance with Weakness Case Acquisition and Iterative Preference Training
Jun Rao, Zepeng Lin, Xuebo Liu +6
Large Language Models (LLMs) often require domain-specific fine-tuning to address targeted tasks, which risks degrading their general capabilities. Maintaining a balance between do…
Pangu Ultra MoE: How to Train Your Big MoE on Ascend NPUs
Yehui Tang, Yichun Yin, Yaoyuan Wang +71
Sparse large language models (LLMs) with Mixture of Experts (MoE) and close to a trillion parameters are dominating the realm of most capable language models. However, the massive…
Pangu Ultra: Pushing the Limits of Dense Large Language Models on Ascend NPUs
Yichun Yin, Wenyong Huang, Kaikai Song +49
We present Pangu Ultra, a Large Language Model (LLM) with 135 billion parameters and dense Transformer modules trained on Ascend Neural Processing Units (NPUs). Although the field…