4 citations · 9 across the 17 of their papers we have counts for
4 papers · 1 filter
Router-Tuning: A Simple and Effective Approach for Enabling Dynamic-Depth in Transformers
Shwai He, Tao Ge, Guoheng Sun +3
Traditional transformer models often allocate a fixed amount of computational resources to every input token, leading to inefficient and unnecessary computation. To address this, t…
Domino: Eliminating Communication in LLM Training via Generic Tensor Slicing and Overlapping
Guanhua Wang, Chengming Zhang, Zheyu Shen +2
Given the popularity of generative AI, Large Language Models (LLMs) often consume hundreds or thousands of GPUs for parallelizing and accelerating the training process. Communicati…
What Matters in Transformers? Not All Attention is Needed
Shwai He, Guoheng Sun, Zheyu Shen +1
While scaling Transformer-based large language models (LLMs) has demonstrated promising performance across various tasks, it also introduces redundant architectures, posing efficie…
Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques
Shwai He, Daize Dong, Liang Ding +1
Scaling large language models has driven remarkable advancements across various domains, yet the continual increase in model size presents significant challenges for real-world dep…