718 citations · 1k across the 44 of their papers we have counts for
5 papers · 1 filter
Turn Waste into Worth: Rectifying Top- Router of MoE
Zhiyuan Zeng, Qipeng Guo, Zhaoye Fei +7
Sparse Mixture of Experts (MoE) models are popular for training large language models due to their computational efficiency. However, the commonly used top- routing mechanism su…
AdaLomo: Low-memory Optimization with Adaptive Learning Rate
Kai Lv, Hang Yan, Qipeng Guo +2
Large language models have achieved remarkable success, but their extensive parameter size necessitates substantial memory for training, thereby setting a high threshold. While the…
What Dense Graph Do You Need for Self-Attention?
Yuxin Wang, Chu-Tak Lee, Qipeng Guo +4
Transformers have made progress in miscellaneous tasks, but suffer from quadratic computational and memory complexities. Recent works propose sparse Transformers with attention on…
Fork or Fail: Cycle-Consistent Training with Many-to-One Mappings
Qipeng Guo, Zhijing Jin, Ziyu Wang +5
Cycle-consistent training is widely used for jointly learning a forward and inverse mapping between two domains of interest without the cumbersome requirement of collecting matched…
Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networks
Minjie Wang, Da Zheng, Zihao Ye +12
Advancing research in the emerging field of deep graph learning requires new tools to support tensor computation over graphs. In this paper, we present the design principles and im…