11 citations · 14 across the 6 of their papers we have counts for
6 papers
veScale: Consistent and Efficient Tensor Programming with Eager-Mode SPMD
Youjie Li, Cheng Wan, Zhiqi Lin +10
Large Language Models (LLMs) have scaled rapidly in size and complexity, requiring increasingly intricate parallelism for distributed training, such as 3D parallelism. This sophist…
Emerging Properties in Unified Multimodal Pretraining
Chaorui Deng, Deyao Zhu, Kunchang Li +9
Unifying multimodal understanding and generation has shown impressive capabilities in cutting-edge proprietary systems. In this work, we introduce BAGEL, an open-source foundationa…
MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router
Yanyue Xie, Zhi Zhang, Ding Zhou +6
Mixture-of-Experts (MoE) architectures face challenges such as high memory consumption and redundancy in experts. Pruning MoE can reduce network weights while maintaining model per…
Parcae: Proactive, Liveput-Optimized DNN Training on Preemptible Instances
Jiangfei Duan, Ziang Song, Xupeng Miao +5
Deep neural networks (DNNs) are becoming progressively large and costly to train. This paper aims to reduce DNN training costs by leveraging preemptible instances on modern clouds,…
Reward Collapse in Aligning Large Language Models
Ziang Song, Tianle Cai, Jason D. Lee +1
The extraordinary capabilities of large language models (LLMs) such as ChatGPT and GPT-4 are in part unleashed by aligning them with reward models that are trained on human prefere…
Learn To Remember: Transformer with Recurrent Memory for Document-Level Machine Translation
Yukun Feng, Feng Li, Ziang Song +2
The Transformer architecture has led to significant gains in machine translation. However, most studies focus on only sentence-level translation without considering the context dep…