57 citations · 161 across the 13 of their papers we have counts for
6 papers · 1 filter
Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism
Xupeng Miao, Yujie Wang, Youhe Jiang +4
Transformer models have achieved state-of-the-art performance on various domains of applications and gradually becomes the foundations of the advanced large deep learning (DL) mode…
CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attention
Ziyu Guo, Renrui Zhang, Longtian Qiu +4
Contrastive Language-Image Pre-training (CLIP) has been shown to learn visual representations with great transferability, which achieves promising accuracy for zero-shot classifica…
OSDP: Optimal Sharded Data Parallel for Distributed Deep Learning
Youhe Jiang, Fangcheng Fu, Xupeng Miao +2
Large-scale deep learning models contribute to significant performance improvements on varieties of downstream tasks. Current data and model parallelism approaches utilize model re…
Towards Communication-efficient Vertical Federated Learning Training via Cache-enabled Local Updates
Fangcheng Fu, Xupeng Miao, Jiawei Jiang +2
Vertical federated learning (VFL) is an emerging paradigm that allows different parties (e.g., organizations or enterprises) to collaboratively build machine learning models with p…
HetuMoE: An Efficient Trillion-scale Mixture-of-Expert Distributed Training System
Xiaonan Nie, Pinxue Zhao, Xupeng Miao +2
As giant dense models advance quality but require large amounts of GPU budgets for training, the sparsely gated Mixture-of-Experts (MoE), a kind of conditional computation architec…
AutoML for Deep Recommender Systems: A Survey
Ruiqi Zheng, Liang Qu, Bin Cui +2
Recommender systems play a significant role in information filtering and have been utilized in different scenarios, such as e-commerce and social media. With the prosperity of deep…